๋ณธ๋ฌธ ๋ฐ”๋กœ๊ฐ€๊ธฐ

๐Ÿ’ปTech/๐Ÿ˜hadoop27

SSLError: Failed to connect. Please check openssl library versions. ๐Ÿšซ ERROR INFO 2020-02-14 17:13:35,457 NetUtil.py:70 - Connecting to https://hdp.hadoop.com:8440/ca ERROR 2020-02-14 17:13:35,460 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:618) ERROR 2020-02-14 17:13:35,461 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details. WARNING 2.. 2020. 2. 14.
Ambari table ์ƒ์„ฑ Ambari ์„ค์น˜ ํ›„ ์‹คํ–‰ํ•˜๊ธฐ ์ „ ๊ด€๋ จ ํ…Œ์ด๋ธ”์„ ์ƒ์„ฑํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ƒ์„ฑํ•˜์ง€ ์•Š๊ณ  ambari-server start ์‹คํ–‰์„ ํ•˜๋ฉด ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค. ambari ์„ค์น˜๊ฐ€ ์™„๋ฃŒ ๋˜์—ˆ์œผ๋ฉด ์•„๋ž˜ ๋ช…๋ น์–ด ์‹คํ–‰ํ•ด์„œ ๊ด€๋ จ ํ…Œ์ด๋ธ”์„ ๋ชจ๋‘ ์ƒ์„ฑํ•ด ์ค๋‹ˆ๋‹ค. mysql -u ambari -p ambari < /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql 2020. 2. 14.
[Atlas] Authentication failed. Cloudera Manager์—์„œ Atlas ์„ค์น˜ ํ›„ ์ธ์ฆ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•˜๋Š”๋ฐ Congifuration์—์„œ ์•„๋ž˜ ์˜ต์…˜์„ ์ฒดํฌํ•ด์ฃผ๋ฉด ์ •์ƒ ๋กœ๊ทธ์ธ๋ฉ๋‹ˆ๋‹ค. 2020. 2. 4.
[SOLVED] two-way ssl authentication failed ๐Ÿšซ ERROR (Ambari) two-way ssl authentication failed. ssl CERTIFICATE_VERIFY_FAILED certificate verify failed and ambari agent heartbeat is not sending heartbeats ๐Ÿ’ก SOLVED: 1. open /etc/ambari-server/conf/ambari.properties 2. remove the property security.server.two_way_ssl = true 2020. 1. 21.
Hue hbase Api Error: TSocket read 0 bytes ๐Ÿ’ก SLOVED hbase.regionserver.thrift.http - enable 2020. 1. 15.
[Hadoop] replication ์ˆ˜ ๋ณ€๊ฒฝ ๋ฐ ํ™•์ธ ๋ฐฉ๋ฒ• hadoop replication ์ˆ˜๋ฅผ ๋ช…๋ น์–ด๋กœ ์„ค์ • ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค 1. ์ ์šฉ hadoop fs -setRep 12 [path] 2. ํ™•์ธ fsck ๋ช…๋ น์–ด๋กœ ์ž…๋ ฅ ํ›„ Agerage block replication์„ ํ™•์ธํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค. hdfs fsck [path] ..................Status: HEALTHY Total size: 27820207400 B Total dirs: 419 Total files: 418 Total symlinks: 0 Total blocks (validated): 418 (avg. block size 66555520 B) Minimally replicated blocks: 418 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-.. 2019. 9. 26.
[Hadoop] Erasure Coding ์ ์šฉ ๋ฐฉ๋ฒ• ์ด๋ ˆ์ด์ € ์ฝ”๋”ฉ (Erasure Coding) ์Šคํ† ๋ฆฌ์ง€์—์„œ ๋ฐ์ดํ„ฐ ์ €์žฅ๊ณต๊ฐ„์˜ ํšจ์œจ์„ฑ์„ ์œ„ํ•ด ์„ค๊ณ„๋œ ๋ฐ์ดํ„ฐ ๋ณต์ œ๋ฐฉ์‹ ์ด๋ ˆ์ด์ € ์ฝ”๋”ฉ์€ ์ด๋ ˆ์ด์ € ์ฝ”๋“œ(Erasure Code)๋ฅผ ์ด์šฉํ•˜์—ฌ ๋ฐ์ดํ„ฐ๋ฅผ ์ธ์ฝ”๋”ฉํ•˜๊ณ , ๋ฐ์ดํ„ฐ ์†์‹ค์‹œ ๋””์ฝ”๋”ฉ ๊ณผ์ •์„ ๊ฑฐ์ณ ์›๋ณธ ๋ฐ์ดํ„ฐ๋ฅผ ๋ณต๊ตฌํ•˜๋Š” ๋ฐ์ดํ„ฐ ๋ณต๊ตฌ ๊ธฐ๋ฒ•์ค‘ ํ•˜๋‚˜ ๋ฐ์ดํ„ฐ ์†์‹ค์‹œ ๋ณต๊ตฌํ•  ์ˆ˜ ์žˆ๋Š” ๊ธฐ๋ฒ•์ผ๋ฟ ๋ฐ์ดํ„ฐ ๋ฐฑ์—…์„ ๋Œ€์ฒดํ•˜๋Š” ๋ฐฉ์‹์ด ์•„๋‹ˆ๋ผ๋Š” ๊ฒƒ์— ์ฃผ์˜ ์ด๋ ˆ์ด์ € ์ฝ”๋”ฉ ์ฒ˜๋ฆฌ ๊ณผ์ • ๋ฐ์ดํ„ฐ ์›๋ณธ์„ n๋“ฑ๋ถ„ํ•˜์—ฌ ๋‚˜๋ˆ” > ์—ฐ์‚ฐ๊ณผ์ •(์ธ์ฝ”๋”ฉ)์„ ํ†ตํ•ด k๊ฐœ ํŒจ๋ฆฌํ‹ฐ ์ƒ์„ฑ > (์ผ๋ถ€ ๋ฐ์ดํ„ฐ ์†์‹ค) > n๊ฐœ์˜ ๋ฐ์ดํ„ฐ๋กœ ๋””์ฝ”๋”ฉ์„ ํ†ตํ•ด ์›๋ณธ ๋ณต๊ตฌ HDFS Erasure Coding HDFS : ํ•˜๋‚˜์˜ ํŒŒ์ผ์„ N๊ฐœ์˜ ๋ธ”๋ก์œผ๋กœ ๋ถ„์‚ฐ ์ €์žฅํ•˜๋Š” ํŒŒ์ผ ์‹œ์Šคํ…œ Fualt-tolerance : ํ•˜๋‚˜์˜ ํŒŒ์ผ์„ ์ด๋ฃจ๊ณ  ์žˆ๋Š” N๊ฐœ.. 2019. 9. 20.
Oozie ๋ช…๋ น์–ด ์ •๋ฆฌ oozie๋ž€ Apache Hadoop ์ž‘์—…์„ ๊ด€๋ฆฌํ•˜๋Š” ์›Œํฌํ”Œ๋กœ์šฐ ์Šค์ผ€์ค„๋Ÿฌ ์‹œ์Šคํ…œ ๋ฐ์ดํ„ฐ ์ถ”์ถœ,๋ณ€ํ™˜,์ ์žฌ ์ž‘์—…์ธ ETL(Extraction Transformation Loading)์„ ์ˆ˜ํ–‰ํ•˜๊ธฐ ์œ„ํ•ด ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. โ—พ coordinator ๋“ฑ๋ก coordinator job์„ ์‹คํ–‰ oozie job -config $job.properties -run โ—พ coordinator ์กฐํšŒ ๋“ฑ๋ก๋˜์–ด ์žˆ๋Š” coordinator job ๋ชฉ๋ก์„ ์กฐํšŒ oozie jobs -jobtype coord โ—พ coordinator update coordinator์˜ ๋‚ด์šฉ(coordinator.xml) ์ˆ˜์ • ํ›„ ์—…๋ฐ์ดํŠธ oozie job -config $job.properties -update $coordinator_job_id) ex) .. 2019. 7. 5.

ํ‹ฐ์Šคํ† ๋ฆฌํˆด๋ฐ”