๋ณธ๋ฌธ ๋ฐ”๋กœ๊ฐ€๊ธฐ

๐Ÿ’ปTech/๐Ÿ’งnifi8

[NiFi] ExecuteSQLRecord postgresql ์‚ฌ์šฉ์‹œ ์ฃผ์˜ auto commit ์˜ต์…˜์ด default๋กœ true์ธ๋ฐ postgresql ๊ฐ™์€ ๊ฒฝ์šฐ true๋กœ ์‚ฌ์šฉ ์‹œ, Heavy Query ์ผ ๊ฒฝ์šฐ ๋ชจ๋‘ ๋ฉ”๋ชจ๋ฆฌ์— ์˜ฌ๋ ค์„œ ๊ฐ€์ ธ์˜ค๊ธฐ ๋•Œ๋ฌธ์— nifi ์žฅ์•  ๋ฐœ์ƒ๋จ (nifi ํ”„๋กœ์„ธ์Šค kill ํ•ด์ค˜์•ผ ํ•จ) ๊ฒฐ๋ก ์€ postgresql์€ auto commit ์˜ต์…˜์„ false๋กœ ์„ค์ •ํ•˜๊ณ  fetch ์‚ฌ์ด์ฆˆ๋ฅผ ํ• ๋‹นํ•ด์„œ ๋ฉ”๋ชจ๋ฆฌ ์ด์Šˆ๋ฅผ ์˜ˆ๋ฐฉํ•  ์ˆ˜ ์žˆ์Œ 2023. 8. 22.
[NiFi] FlowFile "Details" ๊ฐ’ attribute ์ถ”์ถœ ๋ฐฉ๋ฒ• Details ํƒญ์—์„œ Filename, File Size ๋ผ๋Š” ๊ฐ’์„ attribute๋กœ ์ถ”์ถœํ•˜๋Š” ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค. UdateAttribute ํ”„๋กœ์„ธ์„œ๋ฅผ ์ƒ์„ฑํ•˜๊ณ  ์•„๋ž˜์™€ ๊ฐ™์ด ์„ค์ •ํ•ด์ฃผ๋ฉด ํ•ด๋‹น ๊ฐ’์„ ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์†์„ฑ์ด๋ฆ„: my_file_name ์†์„ฑ๊ฐ’: ${filename} ์†์„ฑ์ด๋ฆ„: my_file_size ์†์„ฑ๊ฐ’: ${fileSize} 2023. 4. 21.
[NiFi] content ๋‚ด์šฉ์„ attribute๋กœ ์ €์žฅํ•˜๋Š” ๋ฐฉ๋ฒ• NiFi Processor ์ค‘ ExtractText ํ”„๋กœ์„ธ์„œ์—์„œ ์ •๊ทœ์‹์„ ์‚ฌ์šฉํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค. html ๋‚ด์šฉ์„ attribute์— ์ €์žฅํ•ด์„œ sql์— ์ง‘์–ด๋„ฃ๊ธฐ ์œ„ํ•ด ์ถ”์ถœํ–ˆ์Šต๋‹ˆ๋‹ค. ๐Ÿ‘จ‍๐Ÿ’ป ExtractText์—์„œ ๋ณ€๊ฒฝํ•œ properties ๋‚ด์šฉ Maximum Capture Group Length = 1048576 Enable DOTALL Mode = true Enable Multiline Mode = true body = (.*) 2022. 10. 20.
[NiFi] ORACLE, Impala Data Type ๋น„๊ต NiFI์—์„œ ORACLE ๋ฐ์ดํ„ฐ ์ˆ˜์ง‘ํ•  ๋•Œ ์ •๋ฆฌํ•œ ๋‚ด์šฉ์ž…๋‹ˆ๋‹ค. ๐Ÿ“ Data Type ORACLE Impala CLOB string varchar string number(int) decimal(20,0) number(double) decimal(20,3) timestamp timestamp date timestamp binary_double double โ—พ ExecuteSQLRecord Use Avro Logical Types: true Max Rows Per Flow File: 500000 Fetch Size: 500000 โ—พ Impala DBCP Validation query: select 1 2022. 9. 22.
[NiFi] Groovy-Java๋กœ ์—ฌ๋Ÿฌ ๋‚ ์งœ ๋ฝ‘๋Š” ๋ฐฉ๋ฒ• NiFi์—์„œ ์ œ๊ณตํ•˜๋Š” Groovy Script๋Š” ์ž๋ฐ” ํ˜ธํ™˜์ด ๊ฑฐ์˜ ๋˜๋ฏ€๋กœ ์ž๋ฐ” ์ฝ”๋“œ๋กœ ์ž‘์„ฑํ•˜์—ฌ๋„ ์ž˜ ๋Œ์•„๊ฐ Create ExecuteScript Processor PROPERTIES > Select Groovy > Write Script Body Script Contents import java.text.DateFormat; import java.text.ParseException; import java.text.SimpleDateFormat; import java.util.Calendar; import java.util.Date; //attribute์˜ ๋‚ ์งœ(yyyymmdd) ๊ธฐ์ค€๋ถ€ํ„ฐ -30์ผ๊นŒ์ง€ ์ถ”์ถœ flowFile = session.get(); if(!flowFile) return; dt = flo.. 2022. 9. 22.
[NiFi] global variable(์ „์—ญ๋ณ€์ˆ˜) ์„ค์ • nifi์—์„œ global variable(์ „์—ญ๋ณ€์ˆ˜) ์„ค์ •ํ•˜์—ฌ ์—ฌ๋Ÿฌ Process Group์—์„œ ํ˜„์žฌ ๋‚ ์งœ(yyyyMMdd)๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ์š”๊ฑด์ด ์žˆ์—ˆ๋Š”๋ฐ ๋ฐฉ๋ฒ• ์ฐพ๋Š๋ผ ์—„์ฒญ ๊ณ ์ƒํ–ˆ๋„ค์š” ๐Ÿ˜ญ Variables์™€ Parameter 2๊ฐ€์ง€ ๊ธฐ๋Šฅ์„ ์ œ๊ณตํ•˜๋Š”๋ฐ Parameter์—์„œ NiFi EL(Expression Language)์ด ๋จนํ˜€์„œ ๊ธ€๋กœ๋ฒŒ ๋‚ ์งœ ๋ณ€์ˆ˜๋ฅผ ์ถ”์ถœํ•  ์ˆ˜ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค. Variables์—์„œ๋Š” ${now():format('yyyyMMdd')} ์™€ ๊ฐ™์€ ํ‘œํ˜„์‹์„ ๋ชจ๋‘ String ๋ฌธ์ž ๊ฐ’์œผ๋กœ ์ฝ์–ด์„œ ๋ฐ˜ํ™˜ํ•ด์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์—†๊ณ , Parameter๋Š” EL ํ‘œํ˜„์‹์ด ์‚ฌ์šฉ ๋ถˆ๊ฐ€ํ•˜๋‹ค๊ณ  NiFi doc์— ๋‚˜์™€์žˆ์ง€๋งŒ ์‚ฌ์šฉ์ด ๋˜๋„ค์š”..๐Ÿคจ 1. Parameter์—์„œ EL ์‚ฌ์šฉ ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค. 2. ํ”„๋กœ์„ธ์Šค ๊ทธ๋ฃน์— ํŒŒ๋ผ๋ฏธํ„ฐ ๋งคํ•‘ 3.. 2022. 8. 31.
[NiFi] Impala JDBC DBCPConnectionPool ์‚ฌ์šฉ ์˜ˆ์ œ 1. Impala JDBC Connector ๋‹ค์šด๋กœ๋“œ https://www.cloudera.com/downloads/connectors/impala/jdbc/2-6-17.html 2. NiFi DBCPConnectionPool ์„ค์ • Database Driver Location(s) ์—…๋กœ๋“œํ•œ ํŒŒ์ผ ๊ฒฝ๋กœ ์ž…๋ ฅ /opt/cloudera/parcels/CFM/NIFI/lib/ImpalaJDBC42.jar ๊ฒฝ๋กœ๋ฅผ ์ง€์ •ํ•˜์ง€ ์•Š์„ ๊ฒฝ์šฐ ์•„๋ž˜์™€ ๊ฐ™์€ ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•จ Validation query ์ž…๋ ฅํ•ด์ฃผ๋ฉด ์ข‹์Œ (select 1) Could not initialize class com.cloudera.impala.jdbc42.internal.apache.thrift.transport.TSocket 3. Impala SQ.. 2020. 10. 14.
[NiFi] GetFTP - PutHDFS ์—ฐ๋™ ์˜ˆ์ œ ๊ฐ„๋‹จํ•œ nifi ์‚ฌ์šฉ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค. ftp ์„œ๋ฒ„์—์„œ ๋ฐ์ดํ„ฐ๋ฅผ ๊ฐ€์ ธ์™€ hdfs์— ์—…๋กœ๋“œํ•˜๋Š” ์˜ˆ์ œ์ž…๋‹ˆ๋‹ค. 1. Processor ๋ฒ„ํŠผ์„ ์ž‘์—… ํ™”๋ฉด์— ๋Œ์–ด ๋†“์œผ๋ฉด nifi์—์„œ ์ œ๊ณตํ•˜๋Š” processor๋ฅผ ์„ ํƒํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. (GetFTP, PutHDFS ์ถ”๊ฐ€) 2. ์ถ”๊ฐ€ ํ›„ Configure์—์„œ ์„ค์ •์„ ํ•˜๊ณ  processor ์œ„์— ๋งˆ์šฐ์Šค ์ปค์„œ๋ฅผ ์˜ฎ๊ธฐ๋ฉด ๋ฒ„ํŠผ์ด ๋‚˜์˜ค๋Š”๋ฐ ํด๋ฆญ ํ›„ ๋‘ ๊ฐœ์˜ ํ”„๋กœ์„ธ์„œ๋ฅผ ์—ฐ๊ฒฐํ•ด ์ค๋‹ˆ๋‹ค. GetFTP ํ”„๋กœ์„ธ์„œ์— ์Šค์ผ€์ค„์„ 10์ดˆ๋กœ ์„ค์ •ํ•˜์—ฌ 10์ดˆ๋งˆ๋‹ค ์‹คํ–‰๋˜๊ฒŒ ํ•ด ์ฃผ์—ˆ์Šต๋‹ˆ๋‹ค. ftp ์„œ๋ฒ„์™€ ํŒŒ์ผ ์ •๋ณด ์ž…๋ ฅ (Delete Original ์˜ต์…˜์„ true ์„ค์ •ํ•˜๋ฉด get ์ดํ›„ ์›์ฒœ๊ฒฝ๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ์‚ญ์ œ) hdfs ์„ค์ •ํŒŒ์ผ ์œ„์น˜์™€ ์—…๋กœ๋“œํ•  ๊ฒฝ๋กœ๋ฅผ ์ž…๋ ฅํ•ฉ๋‹ˆ๋‹ค. ํ”„๋กœ์„ธ์Šค ์—ฐ๊ฒฐํ•ด ์ค€ ์ƒํƒœ (๋…ธ๋ž€์ƒ‰ Warn.. 2019. 11. 20.