hadoop - What affects amount of data shuffled in spark -

-

for example im executing queries on spark, , in spark ui can see queries have more shuffle , , shuffle seems amount of data read locally , read between executors.

but im not understanding 1 thing, example query below loaded 7gb hdfs suffle read + shuffled write more 10gb. saw other queries load 7gb hdfs , shuffle 500kb. im not understanding this, can please help? amount of data shuffled not related in data read hdfs? 

select    nation, o_year, sum(amount) sum_profit    ( select    n_name nation, year(o_orderdate) o_year,    l_extendedprice * (1 - l_discount) -  ps_supplycost * l_quantity amount           orders o join       (select l_extendedprice, l_discount, l_quantity, l_orderkey, n_name, ps_supplycost         part p join          (select l_extendedprice, l_discount, l_quantity, l_partkey, l_orderkey,                   n_name, ps_supplycost            partsupp ps join             (select l_suppkey, l_extendedprice, l_discount, l_quantity, l_partkey,                      l_orderkey, n_name                              (select s_suppkey, n_name                  nation n join supplier s on n.n_nationkey = s.s_nationkey                ) s1 join lineitem l on s1.s_suppkey = l.l_suppkey             ) l1 on ps.ps_suppkey = l1.l_suppkey , ps.ps_partkey = l1.l_partkey          ) l2 on p.p_name '%green%' , p.p_partkey = l2.l_partkey      ) l3 on o.o_orderkey = l3.l_orderkey   )profit group nation, o_year order nation, o_year desc;

the shuffle spark’s mechanism re-distributing data it’s grouped differently across partitions. typically involves copying data across executors , machines. pretty clear here shuffled data not dependent on amount of input data. however, depends upon operations perform on input data, leads movement of data across executors( , hence machines). please go through http://spark.apache.org/docs/latest/programming-guide.html#shuffle-operations know , understand why shuffling costly process.

looking @ query have pasted, seems doing lot of join operations (haven't looked deep understand ultimate operation doing). , calls moving data across partitions. problem can handled revisiting query , optimizing same or manipulating or pre-procesing input data in manner leads less movement of data ( ex: colocating data has joined fall in same partition). again, example , have determine use case on works best you.


Comments

Post a Comment

Popular posts from this blog

ios - RestKit 0.20 — CoreData: error: Failed to call designated initializer on NSManagedObject class (again) -

-
this error has shown on many times, seems. prominent answer issue seems restkit 0.20 — coredata: error: failed call designated initializer on nsmanagedobject class however, solution posted doesn't work me. let's start beginning: i want upload file server. want parse response make sure uploaded. if not, try again later. model has boolean variable called nsnumber *senttoserver purpose (in coredata, bool saved nsnumber ) . because don't know whether sending going successful, save entity coredata model first before sending via restkit. when post request successful, server returns json string this: {"id":14,"created":true} notice that, of course, id not set on coredata model yet, both id , created new informations model. here's model: @interface recording : nsmanagedobject @property (nonatomic, retain) nsstring * audiofilepath; @property (nonatomic, retain) nsdate * createdat; @property (nonatomic, retain) nsstring * deviceid; @...
Read more

java - Digest auth with Spring Security using javaconfig -

-
so i'm trying create digest authentication spring following documentation trying translate xml "requirements" in java requirements. let's have xml in docs: <bean id="digestfilter" class= "org.springframework.security.web.authentication.www.digestauthenticationfilter"> <property name="userdetailsservice" ref="jdbcdaoimpl"/> <property name="authenticationentrypoint" ref="digestentrypoint"/> <property name="usercache" ref="usercache"/> </bean> <bean id="digestentrypoint" class= "org.springframework.security.web.authentication.www.digestauthenticationentrypoint"> <property name="realmname" value="contacts realm via digest authentication"/> <property name="key" value="acegi"/> <property name="noncevalidityseconds" value="10"/> </b...
Read more

laravel - PDOException in Connector.php line 55: SQLSTATE[HY000] [1045] Access denied for user 'root'@'localhost' (using password: YES) -

-
my .env: app_env=local app_debug=true app_key=myappkey app_url=http://localhost db_connection=mysql db_host=localhost db_port=3306 db_database=gestiontaller db_username=root db_password='root' cache_driver=file session_driver=file queue_driver=sync redis_host=127.0.0.1 redis_password=null redis_port=6379 mail_driver=smtp mail_host=mailtrap.io mail_port=2525 mail_username=null mail_password=null mail_encryption=null i've tried password ", ' , without, restarting server every time. i've tried changing localhost 172.0.0.1. yes, password correct, running mysql -u root -p introducing password root in prompt can go mysql. users table: +------------------+-----------+ | user | host | +------------------+-----------+ | debian-sys-maint | localhost | | mysql.sys | localhost | | root | localhost | | root1 | localhost | +------------------+-----------+ i've added homestead.yaml this: database_port: 3306 ...
Read more