Welcome to the new Gigaspaces XAP forum. To recover your account, please follow these instructions.

Ask Your Question
0

Gigaspaces log4j kafka elasticsearch

Hi,

I would like to ship gigaspaces logs (GSA, GSC) to elasticsearch, without the use of ELK heavy logstash, but rather using log4j. Final setup CGC/GSA (using log4j, kafka-clients and jackson.core) -> kafka cluster -> Elasticsearch.

Do I only need to provide the additional libraries and replace the gs_logging.properties with a log4j properties file, or is there more to do to achieve this.

Second option would be CGC/GSA (using logback kafka-clients and logstash-encoder) -> elasticsearch. Either would be fine.

Thank you.

asked 2018-02-07 11:18:49 -0500

C45p31 gravatar image
edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
0

Figured out how to make it happen.

answered 2018-02-28 05:57:06 -0500

C45p31 gravatar image
edit flag offensive delete link more
0

Hi

If you want GigaSpaces to use log4j you have to tell it to do so.

handlers=java.util.logging.ConsoleHandler, com.gigaspaces.logger.RollingFileHandler

The default uses our logger; you should set it to your log4j logger.

Look here if you have further questions:

https://docs.gigaspaces.com/xap/12.2/...

Regards

Yuval

answered 2018-02-08 07:08:55 -0500

Yuval gravatar image
edit flag offensive delete link more

Comments

Hi Yuval,

Thank you for your answer.

I have tried, but with logback, without success. Strange enough, after changing the logging properties, the JVM even ignores the -Xloggc switch and prints in stdout:

java -server -d64 -verbose:gc -XX:+AggressiveOpts -XX:+HeapDumpOnOutOfMemoryError -XX:ParallelGCThreads=8 -XX:ParallelCMSThreads=4 -XX:MaxTenuringThreshold=8 -XX:SurvivorRatio=5 -XX:PrintCMSStatistics=1 -XX:CMSWaitDuration=60000 -XX:CMSTriggerInterval=600000 -XX:+PrintReferenceGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+CMSScavengeBeforeRemark -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -Xloggc:/${SOME_PATH_HERE}/gc_LUS-gsa_20180227192113.log -Dcom.gs.zones=LUS -Dcom.gs.transport_protocol.lrmi.use_async_connect=false -Dcom.gs.transport_protocol.lrmi.bind-port=9001-9017 -Dcom.gs.multicast.enabled=false -Dcom.gs.multicast.discoveryPort=9000 -Dcom.gigaspaces.system.registryPort=9018 -Dcom.sun.jini.reggie.initialUnicastDiscoveryPort=9000 -Dcom.gs.jini_lus.unicast_discovery.enabled=true -Dcom.gigaspaces.unicast.interval=5000,10000 -Dcom.gigaspaces.system.registryRetries=10 -Dcom.gigaspaces.start.httpPort=9020 -Dcom.gigaspaces.start.hostAddress= -Dcom.gs.embedded-services.httpd.port=9021 -Dcom.gs.browser.unicast_discovery=host:port,host:port -Dcom.gs.manager.rest.port=8595 -Dcom.gs.lrmi.filter.factory=com.gigaspaces.lrmi.nio.filters.SSLFilterFactory -Dcom.gs.lrmi.filter.security.protocol=TLSv1.2 -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog -Dweb.ssl.keyStorePassword.envvar=secret -Xms512m -Xmx512m -DagentId= -DgsaServiceID= -DenableDynamicLocators= -cp :${SOME_PATH_HERE}/XAP/lib/optional/sigar/:${SOME_PATH_HERE}/XAP:${SOME_PATH_HERE}/XAP/lib/required/xap-common.jar:${SOME_PATH_HERE}/XAP/lib/platform/service-grid/xap-service-grid-boot.jar:${SOME_PATH_HERE}/XAP/lib/optional/kafka/ -Dlogback.configurationFile=file:${SOME_PATH_HERE}/XAP/config/log/xap_logging.properties -Djava.security.policy=${SOME_PATH_HERE}/XAP/policy/policy.all -Djava.rmi.server.hostname=pgtapi11 -Dcom.gs.home=${SOME_PATH_HERE}/XAP com.gigaspaces.start.SystemBoot services=GSA gsa.global.lus 0 gsa.global.gsm 0 gsa.gsc 0 gsa.gsm_lus 0 gsa.gsm 0 gsa.lus 1

Here are my jars: user@server:~$ ls ${SOME_PATH}/XAP/lib/optional/kafka/ activation-1.1.jar jackson-core-2.8.9.jar jansi-1.9.jar logback-classic-1.2.3.jar logstash-logback-encoder-4.11.jar slf4j-api-1.7.25.jar commons-compiler-3.0.6.jar jackson-databind-2.8.9.jar javax.servlet-api-3.1.0.jar logback-core-1.2.3.jar lz4-java-1.4.jar snappy-java-1.1.4.jar jackson-annotations-2.8.0.jar janino-3.0.6.jar kafka-clients-1.0.0.jar logback-kafka-appender-0.2.0-RC1.jar mail-1.4.jar

And here is the logging properties:

<configuration> <appender name="FILE" class="ch.qos.logback.core.ConsoleAppender"> <encoder> <charset>UTF-8</charset> <pattern>%date %level [%thread] %logger{10} [%file:%line] %X{LRID}- %msg%n%ex</pattern> </encoder> </appender> <appender name="kafka" class="com.github.danielwegener.logback.kafka.KafkaAppender"> <encoder class="net.logstash.logback.encoder.LogstashEncoder"> <customfields>{"environment":"TEST"}</customfields> </encoder> <topic>primegate-primegate-gsa</topic> <keyingstrategy class="com.github.danielwegener.logback.kafka.keying.NoKeyKeyingStrategy"/> <deliverystrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"/> <appendtimestamp>true</appendtimestamp> <producerconfig>bootstrap.servers=host:port,host:port</producerconfig> <producerconfig>acks=0</producerconfig> <producerconfig>linger.ms=1000</producerconfig> <producerconfig>max.block.ms=0</producerconfig> <producerconfig>client.id=SOME_ID</producerconfig> </appender> <root> <level value="DEBUG"/> <appender-ref ref="kafka"/> <appender-ref ref="FILE"/> </root> </configuration>

It starts, but I only get the GC output ...(more)

C45p31 gravatar imageC45p31 ( 2018-02-28 03:52:57 -0500 )edit

If I just put in the original XAP loggin properties file the new logger: handlers=com.gigaspaces.logger.RollingFileHandler, com.github.danielwegener.logback.kafka.KafkaAppender

I get this:

Can't load log handler "com.github.danielwegener.logback.kafka.KafkaAppender" java.lang.ClassCastException: com.github.danielwegener.logback.kafka.KafkaAppender cannot be cast to java.util.logging.Handler java.lang.ClassCastException: com.github.danielwegener.logback.kafka.KafkaAppender cannot be cast to java.util.logging.Handler Can't load log handler "com.github.danielwegener.logback.kafka.KafkaAppender" java.lang.ClassCastException: com.github.danielwegener.logback.kafka.KafkaAppender cannot be cast to java.util.logging.Handler

C45p31 gravatar imageC45p31 ( 2018-02-28 04:06:01 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

Stats

Asked: 2018-02-07 11:18:49 -0500

Seen: 110 times

Last updated: Feb 28