janbalik Software, Support, and Training for Apache Cassandra en-US Sun, 26 May 2013 03:30:20 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/cassandra-performance-split-or-not-to-split-cf#post-9139 Fri, 22 Feb 2013 19:35:04 +0000 arviarya 9139@http://www.datastax.com/support-forums/ <p>You can split a column family on requirement basis. It depends how you are organizing your data or fetch. Generally, huge columns columns families may throw lots of data in your program that you may not need.<br /> Better organise your data in single and multiple CFs as per requirement. There are advantage of split CFs if you do not need to read both/all CFs for fetch.<br /> Writes are faster, hence you can send few duplicate data in both CFs to avoid joins. </p> http://www.datastax.com/support-forums/topic/why-a-3rd-column-family-when-creating-a-custom-index-on-cassandra#post-8850 Fri, 01 Feb 2013 17:00:12 +0000 Janbalik 8850@http://www.datastax.com/support-forums/ <p>Since I think someone else could have the same doubt than me, I'm going to answer my own question with the things I have found:</p> <p>As I supposed the main problem is the concurrency. If we assume that many users at the same time can be changing the same indexed value, as you have to read the index before updating, between the time you read a value and the time you update a value in the index another user could have changed again that value. As well as, from the moment the value is updated until you update the index the system could crash. Then, after a few concurrent changes the index could have old values that point to rows that have not that value. </p> <p>By adding the third column family this process is safer but NOT 100% SAFE. </p> <p>And a last thing: from my understanding, if there is no concurrency when updating the values, then there must be no problem. Let's supose you are indexing some user data. If only the owner of the data is allowed to modify the data, there is no concurrency at all. The unique risk is the system crashes before you finish the process to align the index with the value, but this operation is idempotent, so you can repeat it until success. </p> <p>Hope this explain what I have understood and help others. </p> http://www.datastax.com/support-forums/topic/cassandra-performance-split-or-not-to-split-cf#post-8849 Fri, 01 Feb 2013 16:40:07 +0000 Janbalik 8849@http://www.datastax.com/support-forums/ <p>Hi @ll,</p> <p>I'm working on the design of a Cassandra database to learn about it. But I have a question I would like some expert help me to clarify:</p> <p>I have read that the rows of each column family are distributed through the nodes, thus each node has a part of the rows of a given column family. Does it mean that it is not a good idea to divide a column family into many column families even when that column family has millions of rows?</p> <p>My experience with RDBMS says that is better to split very big tables into smaller tables to get a better performance, but it seems that in Cassandra there is no need of this and, even more, if I have many column families I would need more memory. Am I right? Is it better keeping many rows in a column family to get a better performance than split the column family in many?</p> <p>Thanks! </p> http://www.datastax.com/support-forums/topic/why-a-3rd-column-family-when-creating-a-custom-index-on-cassandra#post-8702 Sun, 27 Jan 2013 10:54:22 +0000 Janbalik 8702@http://www.datastax.com/support-forums/ <p>Hi Khahn, </p> <p>Don't say sorry friend ;) You are helping me! Thank you very much. I'm still tring to understand this. I think a key is this: "Updates to the index an data are not atomic". Another one is "a node might index data held by another node". This is aligned with what I thought: you can have an index to a data which does not exist or a data not indexed. But, why using a third with the values indexed we can avoid this problem? We could say updates to the index, data and values indexes are not atomic, I mean, from my understanding, there could be concurrency and consistency problems also between the third CF (values indexed) and data. Or am I wrong? If so, why? Can you (or anyone) explain me why I'm wrong? </p> <p>Thanks!! </p> http://www.datastax.com/support-forums/topic/why-a-3rd-column-family-when-creating-a-custom-index-on-cassandra#post-8617 Thu, 24 Jan 2013 02:09:59 +0000 khahn 8617@http://www.datastax.com/support-forums/ <p>Sorry the link didn't answer your question. I learned a little more after researching your question, but I'm afraid it's not a complete answer. I hope an expert gives you a better explanation soon. </p> <p>The third column family is needed because simply reading the previous value from the users CF before updating it and then removing the index entry for that value from the Users_Index_Entries will not reliably work due to Cassandra's model of eventual consistency and lack of transactions. When creating your own index, a node might index data held by another node. Updates to the index and data are not atomic. To overcome these problems, you maintain a third column family to list previous values for the property of an item and use the list to remove these values from the index before adding the new value to the index. </p> http://www.datastax.com/support-forums/topic/why-a-3rd-column-family-when-creating-a-custom-index-on-cassandra#post-8609 Wed, 23 Jan 2013 18:10:30 +0000 Janbalik 8609@http://www.datastax.com/support-forums/ <p>Hi Khahn, </p> <p>Thanks for your answer. I had read the links you recommend. And I have read them again (a few times), but I still don't understand the problem. Probably it is obvious, but I don't see it. Why there could be a concurrency issue if I read from users CF instead of reading from Users_Index_Entries? Both CF must have the most recent value due to the timestamp which is part of each column. What is the difference? Following the process I don't see why the value from Users CF could be not aligned with the index CF if I use only these 2 CFs. Do you (or anyone) know what is the problem that could happen? Have any example of the issue?</p> <p>Thank you very much! </p> http://www.datastax.com/support-forums/topic/why-a-3rd-column-family-when-creating-a-custom-index-on-cassandra#post-8582 Tue, 22 Jan 2013 19:34:12 +0000 khahn 8582@http://www.datastax.com/support-forums/ <p>Does the last paragraph (below) of this InfoQ article help: <a href="http://www.infoq.com/news/2011/07/cassandraindexing" rel="nofollow">http://www.infoq.com/news/2011/07/cassandraindexing</a> This article also contains a link to sample code: <a href="https://github.com/edanuff/CassandraIndexedCollections" rel="nofollow">https://github.com/edanuff/CassandraIndexedCollections</a></p> <p>"In general, he (Anuff) said that reading before writing can be an issue with Cassandra. Rather than doing locking (e.g., with ZooKeeper), Anuff presented a technique that uses three Column Families. For example, in a table with a users Column Family and an indexes Column Family, there will be a third Column Family Users_Index_Entries. Updates first read the previous index values from this column family to avoid concurrency issues and both it and Users use timestamped columns to avoid the need for locking. ..." </p> http://www.datastax.com/support-forums/topic/why-a-3rd-column-family-when-creating-a-custom-index-on-cassandra#post-8559 Mon, 21 Jan 2013 19:24:51 +0000 Janbalik 8559@http://www.datastax.com/support-forums/ <p>As I always say, sorry for my English. I'm working on creating some manual indexes for some column families in Cassandra. I have read everything I could about this but I have found something I'm not able to understand properly.</p> <p>In this presentation (<a href="http://www.slideshare.net/edanuff/indexing-in-cassandra)" rel="nofollow">http://www.slideshare.net/edanuff/indexing-in-cassandra)</a>, pages 36 to 45, done by Ed Anuff, I have seen his simple example for creating an index for a Users column family. He uses the 2 obvious CFs and another one to deal with concurrency. This third CF is "my problem". If I'm not wrong, Cassandra will always store the most recent value for each column. If this value is indexed, I have to update it in the Index CF (delete old index and create the new one), but why it is necessary the third CF? When I think about that and the concurrency, what my understanding says is: ok, many people updating a value which is indexed. It will mean a lot of work updating the index, but finally the last value will be in the Users CF and also in the Index CF, that's why there is a timestamp per column, so what's the matter with the concurrency? Even more, if the value can be updated only by one user (the owner of the data), there will be no concurrency...</p> <p>I know I am a big ignorant in Cassandra affairs, but I don't see the reason behind the third CF. Ed Anuff explains that using this third column family you can restore the indexes to a consistent status (<a href="http://www.anuff.com/2010/07/secondary-indexes-in-cassandra.html)" rel="nofollow">http://www.anuff.com/2010/07/secondary-indexes-in-cassandra.html)</a>, but, why are them going to fall into an inconsistent status? And, if this happens, the Users CF could be enough to restore the index, or am I wrong?</p> <p>Please, could someone explain me this? What is/are my error/s? </p> http://www.datastax.com/support-forums/topic/better-choice-on-manual-indexes#post-8402 Mon, 14 Jan 2013 13:42:09 +0000 Janbalik 8402@http://www.datastax.com/support-forums/ <p>Hi khahn! I have checked the links and I had already read them before writting my post, except the last one. Unfortunately, I think secondary indexes are not the best solution and the example of the last link I think it is not suitable for my needs, but I appreciate a lot you help ;). Thank you very much. </p> http://www.datastax.com/support-forums/topic/better-choice-on-manual-indexes#post-8390 Sun, 13 Jan 2013 19:24:52 +0000 Janbalik 8390@http://www.datastax.com/support-forums/ <p>Thank you very much, khahn ;) I'm going to check all the links with a lot of interest! </p> http://www.datastax.com/support-forums/topic/better-choice-on-manual-indexes#post-8389 Sun, 13 Jan 2013 17:45:46 +0000 khahn 8389@http://www.datastax.com/support-forums/ <p>Thanks for the interesting question. I cannot provide a complete answer, but hope to give you a clue until an expert answers. Please see:</p> <p><a href="http://www.datastax.com/docs/1.2/ddl/indexes#when-to-use-secondary-indexes" rel="nofollow">http://www.datastax.com/docs/1.2/ddl/indexes#when-to-use-secondary-indexes</a><br /> <a href="http://www.datastax.com/docs/1.2/ddl/indexes#when-not-to-use-secondary-indexes" rel="nofollow">http://www.datastax.com/docs/1.2/ddl/indexes#when-not-to-use-secondary-indexes</a><br /> <a href="http://www.datastax.com/docs/1.2/ddl/indexes#building-and-using-secondary-indexes" rel="nofollow">http://www.datastax.com/docs/1.2/ddl/indexes#building-and-using-secondary-indexes</a>, which describes the "tables as indexes" (same as your second option)<br /> <a href="http://www.datastax.com/docs/1.2/ddl/table#example-a-music-service" rel="nofollow">http://www.datastax.com/docs/1.2/ddl/table#example-a-music-service</a>, which covers using a secondary index in the songs example. </p> http://www.datastax.com/support-forums/topic/better-choice-on-manual-indexes#post-8383 Sun, 13 Jan 2013 13:04:23 +0000 Janbalik 8383@http://www.datastax.com/support-forums/ <p>Another detail: I have calculated that using the second option (storing all the attributes in the CF which works as "indexes") I need about 80% more memory than using the first option (CFs really work as indexes to find the right data in the "main" CF of songs). </p> http://www.datastax.com/support-forums/topic/better-choice-on-manual-indexes#post-8382 Sun, 13 Jan 2013 11:58:19 +0000 Janbalik 8382@http://www.datastax.com/support-forums/ <p>Hi @ll experts,</p> <p>First, excuse me for my English. It is not my native language. I'm working on moving a SQL database to Cassandra but I have a question I'm not able to solve. Let's say I have a SQL table where I store songs. Each song has an ID as primary key which allows to access all its related data, which are stored in the fields of the row given by the key. I also have some indexes to search using some different criterias as the author, gender, title... </p> <p>When I think on moving this to a Cassandra schema, I work around the idea that I can create an equivalent column family, where the song ID is the row key and the song attributes are the columns. Then, I can create 5 or 6 manual indexes to search by author, title, gender and more. The author, title... will be the column key (adding some extra data to keep them unique, using a composite column name) and the value will be the song ID for searching in the static column family where each row is identified by the song ID.</p> <p>But I here appears my doubt. What is better: each index CF storing only the ID or storing all the attributes? The first option allows me to reduce the amount of necessary memory, but I need (at least) 2 reads to get each song attributes. With the second option I need more memory because repeat the same information once per index, but by in one read I can get all the attributes I need. I think I can assume the extra memory needed if this will be a faster schema, but, it will be really faster? Having a bigger database will not make it work slower? Or the slower operation is to search each row given by the index CF due to the way Cassandra stores the rows and due to the 2 reads?</p> <p>Any help will be very appreciated. </p> <p>Thanks in advance! </p>