spark-2015-from-databricks-pdf

GOAL: unified engine across data sources, workloads and environments.

Highlights: dataframes (1.3), SparkR (1.4), …

See all video and slides at http://spark-summit.org

Posted in Me.

“Hortonworks, IBM and Pivotal begin shipping standardized Hadoop. The standardization effort is part of the Open Data Platform initiative, which is an industry effort to ensure all versions of Hadoop are based on the same Apache core..”. Read all the full zdnet.com article

This is t

Posted in Me.

Manually

essbase@olap-server:~> /opt/essbase/Oracle/Middleware/EPMSystem11R1/products/Essbase/EssbaseServer/templates/startMaxl.sh

 Essbase MaxL Shell 64-bit - Release 11.1.2 (ESB11.1.2.3.505B018)
 Copyright (c) 2000, 2014, Oracle and/or its affiliates.
 All rights reserved.

MAXL> login Hypadmin mypassword on olap-server.redaelli.org;

 OK/INFO - 1051034 - Logging in user [Hypadmin@Native Directory].
 OK/INFO - 1241001 - Logged in to Essbase.

MAXL> export database P_BSO.Plan1 level0 data to data_file 'ExpLev0_P_BSO.Plan1';

 OK/INFO - 1054014 - Database Plan1 loaded.
 OK/INFO - 1051061 - Application P_BSO loaded - connection established.
 OK/INFO - 1054027 - Application [P_BSO] started with process id [60396].
 OK/INFO - 1019020 - Writing Free Space Information For Database [Plan1].
 OK/INFO - 1005031 - Parallel export completed for this export thread. Blocks Exported: [2013908]. Elapsed time: [312.35]..
 OK/INFO - 1005002 - Ascii Backup Completed. Total blocks: [2.01391e+06]. Elapsed time: [312.35]..
 OK/INFO - 1013270 - Database export completed ['P_BSO'.'Plan1'].

Automatically

/opt/essbase/Oracle/Middleware/EPMSystem11R1/products/Essbase/EssbaseServer/templates/startMaxl.sh -u Hypadmin -p mypassword -s localhost backup-databases.msh

with a file backup-databases.msh like

export database P_BSO.Plan1 level0 data to data_file 'ExpLev0_P_BSO.Plan1';

 

BUT if you need to export both metadata and data, you should run the command

MAXL> alter database P_BSO_D.Plan1 force archive to file 'P_BSO_D.Plan1.arc';

 

Posted in Me.

What is the best linux distribution for Raspberry PI2? I started with Raspian (Debian is my preferred Linux distribution for servers, desktops and laptops) but docker didn’t work.

But with Archlinux it works fine.

How to create a docker images with Archlinux & RPI2? See http://linucc.github.io/docker-arch-rpi2/

matteoredaelli/docker-karaf-rpi is the first docker image I have created.

Below my docker info output:

[root@raspi1 ~]# docker info
Containers: 4
Images: 9
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 17
Execution Driver: native-0.2
Kernel Version: 3.18.10-1-ARCH
Operating System: Arch Linux ARM
CPUs: 1
Total Memory: 432.8 MiB
Name: raspi1
ID: UOFM:E7CP:2OTL:VTTM:QRP2:JNJ7:UFCI:2MDE:AOYQ:MGTM:Q25F:FL37
WARNING: No memory limit support
WARNING: No swap limit support

Below some docker survival commands:

docker run -i -t --name karaf \
           -p 1099:1099 -p 8101:8101 \
           -p 44444:44444 -v /apps/karaf-deploy:/deploy \
           matteoredaelli/karaf-docker-rpi /bin/bash
docker start karaf
docker stop karaf
docker exec -it karaf bash
docker top
docker ps
docker ps -a
docker images


Posted in Me.

Bigdata projects can be very expensive and can easily fail: I suggest to start with a small, useful but not critical project. Better if it is about unstructured data collection and batch processing. In this case you have time to get practise with the new technologies and the Apache Hadoop system can have not critical downtimes.

At home I have the following system running on a small Raspberry PI: for sure it is not fast ;-)

At work I introduced Hadoop just few months ago for collecting web data and generating daily reports.

Competitor_Analysis_BigData

Posted in Me.

drill

The trend of these years has been switching from SQL (RDBMS) databases to NoSQL databases like Hadoop, MongoDB, Cassandra, Riak, …

SQL is a old but easy and fast way to query data. And people STILL look at it for quering Hadoop and bigdata:

Read details from 10 ways to query hadoop with sql ..

Posted in Me.