๋ณธ๋ฌธ ๋ฐ”๋กœ๊ฐ€๊ธฐ

๐Ÿ’ปTech160

MySQL, MariaDB ํ•œ๊ธ€ ๊นจ์ง ํ˜„์ƒ ํ•ด๊ฒฐ(Hive Metastore) Hive์—์„œ desc๋กœ ํ…Œ์ด๋ธ” ์กฐํšŒ ์‹œ ํ•œ๊ธ€ comment๊ฐ€ ๊นจ์ง€๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ๋Š”๋ฐ, metastore DB ํ•œ๊ธ€ ์„ค์ •์ด ์•ˆ ๋ผ์„œ ๊ทธ๋ ‡์Šต๋‹ˆ๋‹ค. ์•„๋ž˜ ์กฐ์น˜ ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค. MySQL 1. my.cnf ์ˆ˜์ • my.cnf ํŒŒ์ผ ์—ด์–ด์„œ ์•„๋ž˜๋‚ด์šฉ์—์„œ ์—†๋Š”๊ฒƒ๋“ค์„ ์ถ”๊ฐ€ํ•ด ์ฃผ์„ธ์š”. [root@localhost]$ vi /etc/my.cnf ---------------------------------------------------------- [client] character-sets-dir = utf8 [mysqld] init_connect=SET collation_connection = utf8_general_ci init_connect=SET NAMES utf8 character-set-server = utf8 col.. 2023. 8. 21.
[Hive,Impala] sqlํŒŒ์ผ ์‹คํ–‰ํ•  ๋•Œ ๋ณ€์ˆ˜ ๋„˜๊ธฐ๋Š” ๋ฐฉ๋ฒ• โ—พ Hive hive 3.0๋ถ€ํ„ฐ๋Š” hiveconf ์‚ฌ์šฉ์ด ์•ˆ ๋ผ์„œ hivevar๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. -hivevar ์‚ฌ์šฉ hive --hivevar dt=20190923 -f hive.sql -hive.sql ํŒŒ์ผ ๋‚ด์—์„œ ๋ณ€์ˆ˜ ๋ฐ›๋Š” ๋ฐฉ๋ฒ• (ํŒŒํ‹ฐ์…˜ ์ƒ์„ฑ ์˜ˆ์ œ) ALTER TABLE dbnm.tblnm ADD PARTITION(dt='${hivevar:dt}'); โ—พ Impala impala-shell -k --var="dt=20230821" -f impala.sql 2023. 8. 21.
[NiFi] FlowFile "Details" ๊ฐ’ attribute ์ถ”์ถœ ๋ฐฉ๋ฒ• Details ํƒญ์—์„œ Filename, File Size ๋ผ๋Š” ๊ฐ’์„ attribute๋กœ ์ถ”์ถœํ•˜๋Š” ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค. UdateAttribute ํ”„๋กœ์„ธ์„œ๋ฅผ ์ƒ์„ฑํ•˜๊ณ  ์•„๋ž˜์™€ ๊ฐ™์ด ์„ค์ •ํ•ด์ฃผ๋ฉด ํ•ด๋‹น ๊ฐ’์„ ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์†์„ฑ์ด๋ฆ„: my_file_name ์†์„ฑ๊ฐ’: ${filename} ์†์„ฑ์ด๋ฆ„: my_file_size ์†์„ฑ๊ฐ’: ${fileSize} 2023. 4. 21.
[๋ฆฌ๋ˆ…์Šค] JupyterHub ์„ค์น˜ ๋ฐฉ๋ฒ• ๋ฆฌ๋ˆ…์Šค ํ™˜๊ฒฝ (CentOS 7, Python3.8)์—์„œ JupyterHub ์„ค์น˜ ๋ฐฉ๋ฒ• ๊ณต์œ ํ•ฉ๋‹ˆ๋‹ค. ์„ค์น˜ ์ „ Jupyter ์šฉ์–ด ๊ด€๋ จํ•˜์—ฌ ๊ฐ„๋žตํ•˜๊ฒŒ ์ •๋ฆฌ ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. Jupyter Notebook ๋Œ€ํ™”ํ˜• Python Interpreter๋กœ ์›น ํ™˜๊ฒฝ์—์„œ Python ์ฝ”๋“œ ์ž‘์„ฑ ๋ฐ ์‹คํ–‰ํ•˜๋Š” ๊ฐœ๋ฐœ ํ™˜๊ฒฝ(tool) Jupyter Lab Jupyter Notebook์˜ ์ฐจ์„ธ๋Œ€ ๋ฒ„์ „์œผ๋กœ ์‚ฌ์šฉ์ž ํŽธ์˜ ๊ธฐ๋Šฅ๋“ค์ด ์ถ”๊ฐ€๋จ ๋‹ค์ค‘์ฐฝ ์ง€์›, csv/pdf ๋“ฑ ํŒŒ์ผ๋„ ์—ด ์ˆ˜ ์žˆ์–ด์„œ ๋Œ€์‹œ๋ณด๋“œ์ฒ˜๋Ÿผ ์‚ฌ์šฉ ๊ฐ€๋Šฅ JupyterHub ๋ฉ€ํ‹ฐ ์‚ฌ์šฉ์ž ํ™˜๊ฒฝ์—์„œ Jupyter Notebook/Lab์„ ์‚ฌ์šฉ ๐Ÿ“– ์„ค์น˜ ๋ฐฉ๋ฒ• 1. os ํŒจํ‚ค์ง€ ์„ค์น˜ yum install –y nodejs yum install openssl 2. nodejs ํŒจํ‚ค์ง€ ์„ค์น˜ โ—พ.. 2023. 4. 13.
[python] pysqlite3 ์„ค์น˜ ์˜ค๋ฅ˜ ํ•ด๊ฒฐ ๐Ÿšซ ERROR src/connection.h:34:21: fatal error: sqlite3.h: No such file or directory #include "sqlite3.h" compilation terminated. error: command 'gcc' failed with exit status 1 ๐Ÿ’ก SOLVED $ yum install -y libsqlite3x-devel $ pip3 install pysqlite3 2023. 4. 12.
[python] sasl ์„ค์น˜ ์˜ค๋ฅ˜ ํ•ด๊ฒฐ sasl ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ์„ค์น˜ ๊ณผ์ •์—์„œ ์•„๋ž˜์™€ ๊ฐ™์€ ์˜ค๋ฅ˜ ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•๋“ค์ž…๋‹ˆ๋‹ค. ๋ฆฌ๋ˆ…์Šค ํŒจํ‚ค์ง€ ์„ค์น˜๊ฐ€ ํ•„์š”ํ•˜๋„ค์š”. ๊ฒฐ๊ณผ์ ์œผ๋กœ ์•„๋ž˜ ๋ช…๋ น์–ด ์ˆ˜ํ–‰ํ•˜์—ฌ ํŒจํ‚ค์ง€ ์„ค์น˜ํ•ด์ฃผ๋ฉด ๋ฉ๋‹ˆ๋‹ค. ๐Ÿšซ ERROR gcc: error trying to exec 'cc1plus': execvp: No such file or directory sasl/saslwrapper.h:22:23: fatal error: sasl/sasl.h: No such file or directory ๐Ÿ’ก SOLVED $ yum install -y gcc-c++ cyrus-sasl-devel $ pip3 install sasl 2023. 3. 22.
hive/impala udf ๋“ฑ๋ก ๋ฐฉ๋ฒ• hdfs ํŒŒ์ผ ์—…๋กœ๋“œ ํ›„ impala, hive SQL์—์„œ ๊ฐ๊ฐ function ์ƒ์„ฑํ•ด์ฃผ๊ณ , function์ด db ๊ธฐ์ค€์œผ๋กœ ์ƒ์„ฑ๋˜๊ธฐ ๋•Œ๋ฌธ์—, db๋ช…์‹œ๋ฅผ ํ•ด์ค˜์•ผ ํ•ฉ๋‹ˆ๋‹ค. โ—พ Impala create function default.count_date(string) returns string location 'hdfs:///user/hive/udf/udf-0.1.0.jar' symbol='udf.count_date'; โ—พ Hive create function default.count_date as 'udf.count_date' using jar 'hdfs:///user/hive/udf/udf-0.1.0.jar'; 2023. 1. 16.
[Hive] multi delimiter ํ…Œ์ด๋ธ” DDL create external table txt_test( a string, b string, c string, d string, e string ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe' WITH SERDEPROPERTIES ("field.delim"="|\001|") LOCATION 'hdfs://name/tmp/test'; 2023. 1. 11.

ํ‹ฐ์Šคํ† ๋ฆฌํˆด๋ฐ”