Support Forums : DataStax

Increment CounterColumn with hadoop job

(4 posts) (3 voices)

Tags:

No tags yet.

JoelT
Member

Hi all,

I'm trying to write data to cassandra from a hadoop job. I am wondering if it is possible to increment a column with CounterColumnType as default_validation_class using BulkOutputFormat. Maybe I am missing something but it looks like the write method in BulkRecordWriter only leads to addColumn methods.

Thanks for any help on this one!

Posted 8 months ago #
jeromatron
Member

Incrementing counters with Hadoop is probably not a good idea. Hadoop does retries and speculative execution and with counters that would create a mess. If you absolutely must do that, you can just create the raw mutations with thrift in your code, but again, it's probably not a good idea.

Posted 8 months ago #
alexliu
Member

you can switch to ColumnFamilyRecordWriter to create mutation on counter column

Posted 8 months ago #
JoelT
Member

Ok thanks, I agree it is not the good way to do it, with retries and tasks failing it will lead to wrong results. I will compute the whole value before sending it to Cassandra.

Thanks again for the answers

Posted 8 months ago #