gvsafronov/fbx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add import csv function

Browse Source
This commit is contained in:
Kenneth Cheng 2020-05-03 21:06:10 +08:00
parent 79a898fb7b
commit 7e8fc08b04
2 changed files with 151 additions and 33 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

70
README.md
View File

@ -1,6 +1,6 @@
# Redis Module for maintaining hash by simple SQL # Redis Module for maintaining hash by simple SQL (Also provide csv import function)
This module aims to provide simple DML to manipulate the hashes in REDIS for SQL users. It works as simple as you expected. It translates the input statement to a set of pure REDIS commands. It does not need nor generate any intermediate stuffs which occupied your storages. The target data is your hashes only. This module aims to provide simple DML to manipulate the hashes in REDIS for SQL users. It works as simple as you expected. It translates the input statement to a set of pure REDIS commands. It does not need nor generate any intermediate stuffs which occupied your storages. The target data is your hashes only. It also provides the CSV import and export (under construction) function.
## Usage ## Usage
```sql ```sql
@ -51,7 +51,7 @@ If you still have problem in loading the module, please visit: https://redis.io/
## More Examples ## More Examples
#### Select Statement ### Select statement
You may specify multiple fields separated by comma You may specify multiple fields separated by comma
```sql ```sql
127.0.0.1:6379> dbx select name, gender, birth from phonebook 127.0.0.1:6379> dbx select name, gender, birth from phonebook
@ -120,7 +120,7 @@ The above is nearly like REDIS keys command
Each record is exactly a hash, you could use raw REDIS commands ``hget, hmget or hgetall`` to retrieve the same content Each record is exactly a hash, you could use raw REDIS commands ``hget, hmget or hgetall`` to retrieve the same content
#### Where Clause in Select Statement #### Where clause in select statement
Your could specify =, >, <, >=, <=, <>, != or like conditions in where clause. Now the module only support "and" to join multiple conditions. Your could specify =, >, <, >=, <=, <>, != or like conditions in where clause. Now the module only support "and" to join multiple conditions.
```sql ```sql
127.0.0.1:6379> dbx select tel from phonebook where name like Son 127.0.0.1:6379> dbx select tel from phonebook where name like Son
@ -133,7 +133,7 @@ Your could specify =, >, <, >=, <=, <>, != or like conditions in where clause. N
2) "1-888-3333-1412" 2) "1-888-3333-1412"
``` ```
#### Order Clause in Select Statement #### Order clause in select statement
Ordering can be ascending or descending. All sortings are alpha-sort. Ordering can be ascending or descending. All sortings are alpha-sort.
```sql ```sql
127.0.0.1:6379> dbx select name, pos from phonebook order by pos asc 127.0.0.1:6379> dbx select name, pos from phonebook order by pos asc
@ -164,7 +164,7 @@ Ordering can be ascending or descending. All sortings are alpha-sort.
2) "Betty Joan" 2) "Betty Joan"
``` ```
#### Top Clause in Select Statement #### Top clause in select statement
```sql ```sql
127.0.0.1:6379> dbx select top 3 name, tel from phonebook order by pos desc 127.0.0.1:6379> dbx select top 3 name, tel from phonebook order by pos desc
1) 1) name 1) 1) name
@ -183,7 +183,7 @@ Ordering can be ascending or descending. All sortings are alpha-sort.
(empty list or set) (empty list or set)
``` ```
#### Into Clause in Select Statement #### Into clause in select statement
You could create another table by into clause. You could create another table by into clause.
```sql ```sql
127.0.0.1:6379> dbx select * into testbook from phonebook 127.0.0.1:6379> dbx select * into testbook from phonebook
@ -239,7 +239,20 @@ You could create another table by into clause.
10) "F" 10) "F"
``` ```
#### Delete Statement #### Into csv clause in select statement
(The feature is under construction)
```sql
127.0.0.1:6379> dbx select * into csv "/tmp/testbook.csv" from phonebook where pos > 2
(integer) 2
127.0.0.1:6379> quit
$ cat /tmp/testbook.csv
name,tel,birth,pos,gender
"Kenneth Cheng","123-12134-123","2000-12-31","5","M"
"Kevin Louis","111-2123-1233","2009-12-31","6","F"
$
```
### Delete statement
You may also use Insert and Delete statement to operate the hash. If you does not provide the where clause, it will delete all the records of the specified key prefix. (i.e. phonebook) You may also use Insert and Delete statement to operate the hash. If you does not provide the where clause, it will delete all the records of the specified key prefix. (i.e. phonebook)
```sql ```sql
127.0.0.1:6379> dbx delete from phonebook where gender = F 127.0.0.1:6379> dbx delete from phonebook where gender = F
@ -248,8 +261,8 @@ You may also use Insert and Delete statement to operate the hash. If you does no
(integer) 2 (integer) 2
``` ```
#### Insert Statement ### Insert statement
The module provide simple Insert statement which same as the function of the REDIS command hmset. It will append a random string to your provided key (i.e. phonebook). If operation is successful, it will return the key name. The module provides simple Insert statement which same as the function of the REDIS command hmset. It will append a random string to your provided key (i.e. phonebook). If operation is successful, it will return the key name.
```sql ```sql
127.0.0.1:6379> dbx insert into phonebook (name,tel,birth,pos,gender) values ('Peter Nelson' ,1-456-1246-3421, 2019-10-01, 3, M) 127.0.0.1:6379> dbx insert into phonebook (name,tel,birth,pos,gender) values ('Peter Nelson' ,1-456-1246-3421, 2019-10-01, 3, M)
"phonebook:1588298418-551514504" "phonebook:1588298418-551514504"
@ -277,7 +290,42 @@ Note that Redis requires at least one space after the single and double quoted a
127.0.0.1:6379> dbx "insert into phonebook (name,tel,birth,pos,gender) values ('Peter Nelson','1-456-1246-3421','2019-10-01',3, 'M')" 127.0.0.1:6379> dbx "insert into phonebook (name,tel,birth,pos,gender) values ('Peter Nelson','1-456-1246-3421','2019-10-01',3, 'M')"
``` ```
#### Issue command from BASH shell #### From clause for importing CSV file in insert statement
The module provides simple import function by specifying from clause in Insert statement. It only support comma deliminated. Please make sure that the specified import file can be accessed by Redis server.
```bash
$ cat > /tmp/test.csv << EOF
"Kenneth Cheng","123-12134-123","2000-12-31","5","M"
"Kevin Louis","111-2123-1233","2009-12-31","6","F"
EOF
$ redis-cli
127.0.0.1:6379> dbx insert into phonebook (name, tel, birth, pos, gender) from "/tmp/test.csv"
1) "phonebook:1588509697-1579004777"
2) "phonebook:1588509697-1579004778"
127.0.0.1:6379> dbx select name from phonebook
1) 1) name
2) "Kenneth Cheng"
2) 1) name
2) "Kevin Louis"
127.0.0.1:6379> dbx delete from phonebook
(integer) 2
127.0.0.1:6379> quit
$ cat > /tmp/testheader.csv << EOF
name,tel,birth,pos,gender
"Kenneth Cheng","123-12134-123","2000-12-31","5","M"
"Kevin Louis","111-2123-1233","2009-12-31","6","F"
EOF
$ redis-cli
127.0.0.1:6379> dbx insert into phonebook from "/tmp/testheader.csv"
1) "phonebook:1588509697-1579004779"
2) "phonebook:1588509697-1579004780"
127.0.0.1:6379> dbx select name from phonebook
1) 1) name
2) "Kenneth Cheng"
2) 1) name
2) "Kevin Louis"
```
### Issue command from BASH shell
```sql ```sql
$ redis-cli dbx select "*" from phonebook where gender = M order by pos desc $ redis-cli dbx select "*" from phonebook where gender = M order by pos desc
1) 1) "name" 1) 1) "name"

100
src/dbx.c
View File

@ -522,12 +522,20 @@ int SelectCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
return REDISMODULE_OK; return REDISMODULE_OK;
} }
char* trim(char* s, char t) {
char* p = s;
if (p[strlen(s)-1] == t) p[strlen(s)-1] = 0;
if (p[0] == t) p++;
return p;
}
int InsertCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { int InsertCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
if (argc < 2) if (argc < 2)
return RedisModule_WrongArity(ctx); return RedisModule_WrongArity(ctx);
// Table // Table
RedisModuleString *intoKey; char intoKey[128] = "";
RedisModuleString *fromCSV = NULL;
// Process the arguments // Process the arguments
size_t plen; size_t plen;
@ -575,7 +583,7 @@ int InsertCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
break; break;
case -1: case -1:
// parse into clause, assume time+rand always is new key // parse into clause, assume time+rand always is new key
intoKey = RMUtil_CreateFormattedString(ctx, "%s:%u-%i", token, (unsigned)time(NULL), rn++); strcpy(intoKey, token);
step = -2; step = -2;
break; break;
case -2: case -2:
@ -590,6 +598,8 @@ int InsertCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
} }
else if (strcmp("values", token) == 0) else if (strcmp("values", token) == 0)
step = -5; step = -5;
else if (strcmp("from", token) == 0)
step = -7;
else { else {
RedisModule_ReplyWithError(ctx, "values keyword is expected"); RedisModule_ReplyWithError(ctx, "values keyword is expected");
return REDISMODULE_ERR; return REDISMODULE_ERR;
@ -605,8 +615,10 @@ int InsertCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
case -4: case -4:
if (strcmp("values", token) == 0) if (strcmp("values", token) == 0)
step = -5; step = -5;
else if (strcmp("from", token) == 0)
step = -7;
else { else {
RedisModule_ReplyWithError(ctx, "values keyword is expected"); RedisModule_ReplyWithError(ctx, "values or from keyword is expected");
return REDISMODULE_ERR; return REDISMODULE_ERR;
} }
break; break;
@ -618,7 +630,7 @@ int InsertCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
strcpy(stmValue, &token[1]); strcpy(stmValue, &token[1]);
if (token[strlen(token) - 1] == ')') { if (token[strlen(token) - 1] == ')') {
stmValue[strlen(stmValue) - 1] = 0; stmValue[strlen(stmValue) - 1] = 0;
step = 7; step = 8;
} }
else else
step = -6; step = -6;
@ -626,12 +638,16 @@ int InsertCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
else if (token[strlen(token) - 1] == ')') { else if (token[strlen(token) - 1] == ')') {
token[strlen(token) - 1] = 0; token[strlen(token) - 1] = 0;
strcat(stmValue, token); strcat(stmValue, token);
step = 7; step = 8;
} }
else else
strcat(stmValue, token); strcat(stmValue, token);
break; break;
case 7: case -7:
fromCSV = RedisModule_CreateString(ctx, token, strlen(token));
step = 8;
break;
case 8:
RedisModule_ReplyWithError(ctx, "The end of statement is expected"); RedisModule_ReplyWithError(ctx, "The end of statement is expected");
return REDISMODULE_ERR; return REDISMODULE_ERR;
break; break;
@ -647,25 +663,79 @@ int InsertCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
Vector *vField = splitStringByChar(stmField, ","); Vector *vField = splitStringByChar(stmField, ",");
Vector *vValue = splitStringByChar(stmValue, ","); Vector *vValue = splitStringByChar(stmValue, ",");
RedisModule_AutoMemory(ctx);
if (fromCSV != NULL) {
size_t len;
const char *filename = RedisModule_StringPtrLen(fromCSV, &len);
FILE *fp = fopen(filename, "r");
if (fp == NULL) {
RedisModule_ReplyWithError(ctx, "File does not exist");
return REDISMODULE_ERR;
}
char line[1024];
char *value, *field;
size_t n = 0;
RedisModule_ReplyWithArray(ctx, REDISMODULE_POSTPONED_ARRAY_LEN);
while(fgets(line, 1024, fp) != NULL) {
if (line[strlen(line)-1] == 10) line[strlen(line)-1] = 0;
if (line[strlen(line)-1] == 13) line[strlen(line)-1] = 0;
value = strtok(line, ",");
if (Vector_Size(vField) == 0) {
while (value != NULL) {
if (strlen(stmField) > 0) strcat(stmField, ",");
strcat(stmField, trim(value, '"'));
value = strtok(NULL, ",");
}
if (vField) Vector_Free(vField);
vField = splitStringByChar(stmField, ",");
continue;
}
RedisModuleString *key = RedisModule_CreateStringPrintf(ctx, "%s:%u-%i", intoKey, (unsigned)time(NULL), rn++);
for (size_t i=0; i<Vector_Size(vField); i++) {
if (value == NULL) {
RedisModule_ReplyWithError(ctx, "Number of values does not match");
return REDISMODULE_ERR;
}
Vector_Get(vField, i, &field);
RedisModuleCallReply *rep = RedisModule_Call(ctx, "HSET", "scc", key, field, trim(value, '"'));
RedisModule_FreeCallReply(rep);
value = strtok(NULL, ",");
}
n++;
RedisModule_ReplyWithString(ctx, key);
RedisModule_FreeString(ctx, key);
}
fclose(fp);
RedisModule_ReplySetArrayLength(ctx, n);
}
else {
if (Vector_Size(vField) != Vector_Size(vValue)) { if (Vector_Size(vField) != Vector_Size(vValue)) {
RedisModule_ReplyWithError(ctx, "Number of values does not match"); RedisModule_ReplyWithError(ctx, "Number of values does not match");
return REDISMODULE_ERR; return REDISMODULE_ERR;
} }
RedisModule_AutoMemory(ctx);
for (size_t i=0; i<Vector_Size(vField); i++) {
char *field, *value; char *field, *value;
size_t n = 0;
RedisModule_ReplyWithArray(ctx, REDISMODULE_POSTPONED_ARRAY_LEN);
RedisModuleString *key = RMUtil_CreateFormattedString(ctx, "%s:%u-%i", intoKey, (unsigned)time(NULL), rn++);
for (size_t i=0; i<Vector_Size(vField); i++) {
Vector_Get(vField, i, &field); Vector_Get(vField, i, &field);
Vector_Get(vValue, i, &value); Vector_Get(vValue, i, &value);
RedisModuleCallReply *rep = RedisModule_Call(ctx, "HSET", "scc", intoKey, field, value); RedisModuleCallReply *rep = RedisModule_Call(ctx, "HSET", "scc", key, field, trim(value, '"'));
RedisModule_FreeCallReply(rep); RedisModule_FreeCallReply(rep);
} }
n++;
RedisModule_ReplyWithString(ctx, intoKey); RedisModule_ReplyWithString(ctx, key);
RedisModule_FreeString(ctx, intoKey); RedisModule_FreeString(ctx, key);
Vector_Free(vField); RedisModule_ReplySetArrayLength(ctx, n);
Vector_Free(vValue); }
if (vField) Vector_Free(vField);
if (vValue) Vector_Free(vValue);
return REDISMODULE_OK; return REDISMODULE_OK;
} }