Hive support for Cassandra CQL3.

Cassandra:

Apache Cassandra is an open source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. Cassandra offers robust support for clusters spanning multiple datacenters, with asynchronous masterless replication allowing low latency operations for all clients.

Cassandra's data model is a partitioned row store with tunable consistency. Rows are organized into tables; the first component of a table's primary key is the partition key; within a partition, rows are clustered by the remaining columns of the key. Other columns may be indexed separately from the primary key.

Hive:

 Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL.

 CQL3:

 CQL3 (the Cassandra Query Language) provides a new API to work with Cassandra. Where the legacy thrift API exposes the internal storage structure of Cassandra pretty much directly, CQL3 provides a thin abstraction layer over this internal structure. 

 Earlier versions of CQL, CQL2 was close to the concepts of thrift which required clients to decode the CompositeTypes and also exposes the internal storage engine.
Hive support for cassandra:

 The source code for hive support for cassandra is hosted at https://github.com/milliondreams/hive.git under cas-support-cql branch.

Though the code includes support for both CQL3 tables and thrift tables separately by CqlStorageHandler and CassandraStorageHandler, this blog covers using CqlStorageHandler for creating CQL3 tables in cassandra through hive.

 Note: The CqlStorageHandler supports tables created through CQL3 only. CQL2 tables cannot be accessed using this handler.

 Create a CQL3 table through hive:

 Create a database in hive. Then execute the following command.

 hive> CREATE EXTERNAL TABLE test.messages(message_id string, author string, body string) STORED BY 'org.apache.hadoop.hive.cassandra.cql.CqlStorageHandler' WITH SERDEPROPERTIES ("cql.primarykey" = "message_id, author", "comment"="check", "read_repair_chance" = "0.2", "dclocal_read_repair_chance" = "0.14", "gc_grace_seconds" = "989898", "bloom_filter_fp_chance" = "0.2", "compaction" = "{'class' : 'LeveledCompactionStrategy'}", "replicate_on_write" = "false", "caching" = "all")

 where 'test' is the keyspace in cassandra. The above query also creates a column family in cassandra if does not exist. To create a keyspace that doesnot exist in cassandra execute the following query

 hive> CREATE EXTERNAL TABLE test.messages(row_key string, col1 string, col2 string) STORED BY 'org.apache.hadoop.hive.cassandra.cql.CqlStorageHandler' WITH SERDEPROPERTIES  ("cql.primarykey" = "row_key")
TBLPROPERTIES ("cassandra.ks.name" = "mycqlks", "cassandra.ks.stratOptions"="'DC':1, 'DC2':1", "cassandra.ks.strategy"="NetworkTopologyStrategy");

Note: For brevity, only minimal SERDEPROPERTIES are  given in the above query.

 If 'test' keyspace does not exist in cassandra it will be created.

 Inserting values into CQL3 table through hive:

 hive> insert into table messages select * from tweets;

 The values from tweets table are appended to messages table.

Note: With Cassandra INSERT OVERWRITE is same as INSERT INTO as Cassandra merges changes if keys are same.

 Retrieving values from a CQL3 table using hive:

 hive> select * from messages;


References:

Comments

Post a Comment

Popular posts from this blog

Two-way ssl using cURL

cURL - command line tool for transferring data using multiple proto cols.  To es tablish a tw o-way ssl communi cat ion between cURL and a apache tomcat web application, generate a s elf-signed certificate for server and client (machine cURL is running on ). Self-Signed certificate for client: 1. Create a private key for client.      openssl genrsa -out clientprivatekey.pem 2048 2. Create an openSSL self-signed certificate for the client using the private key      openssl req -new -x509 -key clientprivatekey.pem -out clientcert.pem -days 365.      Self-Signed certificate for server: 1. Generate a private key for server openssl genrsa -out serverprivatekey.pem 2048 2.Create an openSSL self-signed certificate for the server using the private key       openssl req -new -x509 -key serverprivatekey.pem -out servercert.pem -days 1095             This prompts you to enter a few pieces of information, use “.” to leave the field blank. When prompted for '
Read more

Two-way SSL authentication on tomcat using OpenSSL self signed certificates.

Two-way SSL authentication: Openssl is used for creating private keys and certificates. Setting up two-way ssl authentication on tomcat is done as follows  Generate a self-signed certificate for tomcat web application  Generate a self-signed certificate for the client (consider browser for this example)  Import client certificate into server's keystore - as CA is not used and server needs to know public key of client.(Optional)  Configuring tomcat's server.xml Generating a private key using openSSL openssl genrsa -out privkey.pem 2048 This generates an RSA private key of 2048 bits. With OpenSSL, the private key contains the public key information as well, so a public key doesn't need to be generated separately. Generating a self-signed certificate using openSSL openssl req -new -x509 -key privkey.pem -out cacert.pem -days 1095 Generates a self-signed certificate that has public key in it valid for 1095 days. 1. Generate a self
Read more