Cloud Computing/Spark
Spark-EC2 Too large frame
MJSon
2017. 10. 2. 16:57
Spark-EC2 Too large frame
spark-ec2 부분을 수정하여 Instance Storage 사이즈 늘림
if opts.spot_price is not None:
# Launch spot instances with the requested price
ondemand_price = get_instance_price(opts.instance_type)
bid_spot_price = ondemand_price if opts.spot_price > ondemand_price else ondemand_price
print("Requesting %d slaves as spot instances with price $%.3f" %
(opts.slaves, bid_spot_price))
zones = get_zones(conn, opts)
num_zones = len(zones)
i = 0
my_req_ids = []
for zone in zones:
num_slaves_this_zone = get_partition(opts.slaves, num_zones, i)
block_map = BlockDeviceMapping() ## 추가
device = EBSBlockDeviceType(); ## 추가
device.size='64' ## 추가 && 여기서 숫자를 바꿔주면 된다.
device.volume_type="standard" ## Standard 가 EBS를 뜻한다.
device.delete_on_termination= True ## 추가
block_map["/dev/xvda"]=device ## 현재 device 이름이 xvda이다.
slave_reqs = conn.request_spot_instances(
price=bid_spot_price,
image_id=opts.ami,
launch_group="launch-group-%s" % cluster_name,
placement=zone,
count=num_slaves_this_zone,
key_name=opts.key_pair,
security_group_ids=[slave_group.id] + additional_group_ids,
instance_type=opts.instance_type,
block_device_map=block_map,
subnet_id=opts.subnet_id,
placement_group=opts.placement_group,
user_data=user_data_content,
instance_profile_name=opts.instance_profile_name)
my_req_ids += [req.id for req in slave_reqs]
i += 1
ebs size 확인
Instance Storage Size를 늘렸지만 Error가 뜬다
오류 화면
Remote Shuffle 이 2GB를 넘을 수 없는 거랑 관련있어보임
우선 16000까지 block size를 최대한으로 하여 실험 돌릴예정