node.js - Understanding Kafka Partition Metadata -

-

i usign kafka-node within nodejs application create topics via loadmetadatafortopics option. want application dynamically understand number of partitions avaiable can distribute messages across partitions.

in single node kafka instance method creating topics , returning metadata this:

  "step1_channelout": {     "0": {       "topic": "step1_channelout",       "partition": 0,       "leader": 1,       "replicas": [         1       ],       "isr": [         1       ]     }   },

however in 3 node cluster, method creates more entries:

{     "0": {         "topic": "step1_channelout",         "partition": 0,         "leader": 3,         "replicas": [             3,             2,             1         ],         "isr": [             3,             2,             1         ]     },     "1": {         "topic": "step1_channelout",         "partition": 1,         "leader": 1,         "replicas": [             1,             3,             2         ],         "isr": [             1,             3,             2         ]     },     "2": {         "topic": "step1_channelout",         "partition": 2,         "leader": 2,         "replicas": [             2,             1,             3         ],         "isr": [             2,             1,             3         ]     },     "3": {         "topic": "step1_channelout",         "partition": 3,         "leader": 3,         "replicas": [             3,             1,             2         ],         "isr": [             3,             1,             2         ]     } }

in case did create 4 partitions? looks me - since last case scenario (really set partitions explicitly) dont care long predictable. said more control have better.

what relationship between topic information in zookeeper versus on kafka server? there bettet way manipulate (create / configure topics) kafka cluster via nodejs?

why 4 partitions? understand three, or one, four?

the way kafka-node works, creates topics based on global kafka configuration found in server.properties. check values of:

num.partitions=12 default.replication.factor=1

there's no automatic relation between number of brokers , number of partitions. can have 100 broker setup want 1 partition topics, or can have single broker setup 1,000 partitions. not related.

there's no non-java api creating topics -- @ least not yet. see unanswered question here.

if want more control on how topics created, still want kafka-node, going have exec command this:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 12 --topic <topic_name_here>

i in node with:

const exec = require('child_process').exec;  function createtopic(topic, replfactor, numpartitions, cb) {   var zkhost = "localhost:2181";   var kafkahome = "/usr/local/kafka";    exec(     `${kafkahome}/bin/kafka-topics.sh --create --zookeeper ${zkhost} -- replication-factor ${replfactor} --partitions ${numpartitions} --topic ${topic}`,     (error,stderr,stdout) => cb(topic)   ); }

Comments

Post a Comment

Popular posts from this blog

ios - RestKit 0.20 — CoreData: error: Failed to call designated initializer on NSManagedObject class (again) -

-
this error has shown on many times, seems. prominent answer issue seems restkit 0.20 — coredata: error: failed call designated initializer on nsmanagedobject class however, solution posted doesn't work me. let's start beginning: i want upload file server. want parse response make sure uploaded. if not, try again later. model has boolean variable called nsnumber *senttoserver purpose (in coredata, bool saved nsnumber ) . because don't know whether sending going successful, save entity coredata model first before sending via restkit. when post request successful, server returns json string this: {"id":14,"created":true} notice that, of course, id not set on coredata model yet, both id , created new informations model. here's model: @interface recording : nsmanagedobject @property (nonatomic, retain) nsstring * audiofilepath; @property (nonatomic, retain) nsdate * createdat; @property (nonatomic, retain) nsstring * deviceid; @...
Read more

java - Digest auth with Spring Security using javaconfig -

-
so i'm trying create digest authentication spring following documentation trying translate xml "requirements" in java requirements. let's have xml in docs: <bean id="digestfilter" class= "org.springframework.security.web.authentication.www.digestauthenticationfilter"> <property name="userdetailsservice" ref="jdbcdaoimpl"/> <property name="authenticationentrypoint" ref="digestentrypoint"/> <property name="usercache" ref="usercache"/> </bean> <bean id="digestentrypoint" class= "org.springframework.security.web.authentication.www.digestauthenticationentrypoint"> <property name="realmname" value="contacts realm via digest authentication"/> <property name="key" value="acegi"/> <property name="noncevalidityseconds" value="10"/> </b...
Read more

laravel - PDOException in Connector.php line 55: SQLSTATE[HY000] [1045] Access denied for user 'root'@'localhost' (using password: YES) -

-
my .env: app_env=local app_debug=true app_key=myappkey app_url=http://localhost db_connection=mysql db_host=localhost db_port=3306 db_database=gestiontaller db_username=root db_password='root' cache_driver=file session_driver=file queue_driver=sync redis_host=127.0.0.1 redis_password=null redis_port=6379 mail_driver=smtp mail_host=mailtrap.io mail_port=2525 mail_username=null mail_password=null mail_encryption=null i've tried password ", ' , without, restarting server every time. i've tried changing localhost 172.0.0.1. yes, password correct, running mysql -u root -p introducing password root in prompt can go mysql. users table: +------------------+-----------+ | user | host | +------------------+-----------+ | debian-sys-maint | localhost | | mysql.sys | localhost | | root | localhost | | root1 | localhost | +------------------+-----------+ i've added homestead.yaml this: database_port: 3306 ...
Read more