Platform Installation Guide (printable)
This guide provides information on installation planning, configuration, and tuning of the platform for optimum performance and security. It is intended for installers who are building a fully-optimized, production installation.
Additional Resources
- For a searchable and printable version of this guide, see the Platform Installation Guide (printable)
- To learn more about the latest release, see the All Release Notes
- See also these additional resources for installation and configuration:
1 Installation Requirements
1.1 Hardware Requirements
| Application Server |
|
| Document storage |
|
| Database Server |
|
| Database backup |
|
1.2 Software Requirements
1.2.1 Installing MySQL Version 8.0.xx
| Operating System |
| ||||
| MySQL Server |
If you are a platform user, see Configure the MySQL Server.
| ||||
| Libraries | Download the third-party libraries and save them in a folder. You may want to integrate them into the system during the installation.
scrollTolerantForwardOnly=true
|
1.2.2 Installing MySQL Version 5.7.xx
| Operating System |
| ||
| MySQL Server |
If you are a platform user, see Configure the MySQL Server. | ||
| Libraries | Download the third-party libraries and save it in a folder. You may want to integrate them into the system during the installation. |
1.2.3 Required for Basic Platform Functionality
| Mail Server |
An email message should be received at the specified address. If the confirmation message does not arrive, check the mail log to discover the cause.
|
| Cache |
Ehcache is available in the Common Tomcat profile. For more information, see Configuring Ehcache. |
| Web Server |
The Apache web server can be placed in front of the Tomcat appserver to deliver static content more efficiently.
For more information, see Installing and Configuring Apache for Use with the Platform. |
| Java |
The installation and operation of the AgileApps Cloud platform requires Java 8. However, the custom code in Java Class continues to be validated against Java 6 syntax as the instrumentation engine responsible for monitoring resource utilization works only on Java 6 byte codes. |
| Browser Support |
As part of the security processes, the Remember me on this computer option is removed from the browser's login page.
|
1.2.4 Required for Additional Functionality
| OpenOffice (optional) |
| ||
| HornetQ (optional) |
The Messaging Server is an optional platform component that:
Get HornetQ from http://www.jboss.org/hornetq/downloads.html. For more information, see Installing the Messaging Server. | ||
| Charting Libraries (optional) |
Additional libraries are needed to email a chart or a report. For more information, see Install the Chart Handling Libraries. |
1.3 License Requirements
About licenses:
- One platform license is required for each instance of the Application Server.
- A short term license is included in the initial installation.
- An email notification is sent out 30 days before a license is due to expire.
- A grace period extends for an additional seven days after the expiration date.
Daily notifications are sent during that period, until the license is renewed or the grace period ends. - To obtain a long-term license, contact your Software AG representative. You will receive an XML license file.
To install the new license:
- Copy the XML license file to {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN
- Restart tomcat
2 Installation Planning
Platform Hardware Stack Platform Software Stack
2.1 Platform Deployment Options
This section describes deployment options that increase scalability and reliability.
- Multiple Application Servers
- Instances of the platform can be deployed on different servers to improve response times.
- Separate servers for Apache httpd
- When the Apache httpd server is used to serve static content, the Apache httpd instances can be deployed on separate servers, while platform instances run on their own servers. Learn More: Installing and Configuring Apache for Use with the Platform
- Separate Messaging Server
- To further improve performance, the Messaging Server can be installed on a standalone system. Or it can run the same system as an appserver.
- Web App Accelerators
- Web Application acceleration services like those provided by Akamai can also be used to improve response times and throughput. Learn More: Web Application Accelerator.
- Multiple memcached servers
- The MemCache Daemon can also be run on one or more separate servers. In addition to the performance improvement, that configuration makes it somewhat easier to add application servers to scale capacity later. Learn More: Configuring memcached
- Certificate-authentication Server
- Adding a certificate-validation server to the network architecture, allows the platform to be accessed by browsers or by REST APIs without a login step, by supplying an identifying certificate.
- Learn more: Client Certificate Authentication in a private AgileApps Cloud (pdf)
- Separate Database Server
- Running the platform and the Database on different servers is highly recommended for a production deployment. Learn More: Configuring MySQL to Run on a Separate Server
- Separate Servers for Backend Processes
- The backend processes used by the application server can be (and should be) deployed to at at least one "backend server" to minimize response time in the customer-facing server they would otherwise be running on. Beyond that, heavily-used services can be deployed to servers of their own, to maximize performance. Learn More: Managing Backend Services
2.2 Prototype Deployment
The following diagram shows the kind of architecture that is typical for a production system pre-10.11 release and post 10.11 release. With the 10.11 release, AgileApps uses the common Tomcat that comes along with the IS installation and does not use the stand-alone Tomcat anymore. This allows AgileApps access to the flow services on the co-hosted IS. The following images allow you to identify the difference in the architecture for pre and post 10.3 release.
- Architecture for pre-10.16 Release
- Architecture for post-10.16 Release
- Architecture for post-10.16 Release
The architecture is explained as follows:
- Node 1 and Node 2 are the AgileApps server instances available in the front-end.
- Node 3 serves as the backend AgileApps Server instance. The critical backend processes shown here (import, export, and scheduling, which uses quartz) are all run from a single platform instance. However, you can employ additional servers, as load demands.
Learn more: Managing Backend Services. - Document storage (which includes pictures and image files) is managed separately from the database. In the example above, it is housed as part of the backend server instance. The Document storage communicates with all the nodes.
Learn more: See the Document Server section in Service Provider Settings. - The user logs into AgileApps and the request comes to the Load Balancer which distributes the traffic across the web servers.
- The web server can be an Apache server or an Nginx server.
Learn more: Installing and Configuring Apache for Use with the Platform. - A Memcached server reduces response time by caching data in memory. Memcached servers are accessed by all Application servers, backend as well as front-end.
Learn more: Configuring memcached - All the nodes are connected to the primary database server. The primary database is running on its own server for better performance. It can also reside within one of the nodes VM1 or VM2 along with the AgileApps server. The database could be MySQL, Amazon RDS, or so on.
Learn more: Configuring MySQL to Run on a Separate Server - The replicated database is running separately on its own server. This ensures more reliability and for performing read-intensive operations like taking backups, generating reports, and exporting data.
Learn more: Running Reports and Storage Checks On a Replicated Database Server. - Requests that access and update the database, whether coming from a user or a backend process, go to the primary database server.
2.3 Choosing a MySQL Replication Strategy
MySQL supports several replication formats:
- Statement-Based Replication (SBR), which replicates entire SQL statements
- Row-Based Replication (RBR), which replicates only changed rows.
- Mixed-Based Replication (MBR), which is a combination of the two.
With Statement-Based Replication, SQL statements are propagated using the standard statement-based, binary logging format. That is the default replication format in the version of MySQL that ships with the platform.
Row-based binary logging is a mechanism that logs changes in individual table rows. With row-based replication, the master writes events to the binary log that indicate how individual table rows are changed.
When mixed format is in effect, statement-based logging is used by default, but automatically switches to row-based logging when it is necessary to do so.
Learn More:
- Replication: http://dev.mysql.com/doc/refman/5.5/en/replication.html
- Replication formats: http://dev.mysql.com/doc/refman/5.5/en/replication-formats.html
- Comparison of Statement-Based and Row-Based Replication:
http://dev.mysql.com/doc/refman/5.5/en/replication-sbr-rbr.html - How To Set Up Database Replication In MySQL:
http://www.howtoforge.com/mysql_database_replication
2.4 Choosing a MySQL High-Availability Option
MySQL can be configured for High Availability using the following schemes supported by MySQL:
- MySQL Replication
- Replication provides data safety. It also lets you run reports and backups from the secondary server, reducing the load on the primary server and improving the response time seen by users. There is some exposure to data loss, since recently-seen data may not have been replicated when a failover occurs--but on the other hand, transactions don't incur the lag time added by serialized (synchronous) replication of data.
- MySQL Replication + Heartbeat
- Using Linux HA capabilities, MySQL can be configured for automatic IP-address failover.
- Learn More: http://dev.mysql.com/doc/refman/5.5/en/ha-overview.html
- DRBD + MySQL Heartbeat
- While the previous two schemes do statement-based replication, Distributed Replicated Block Devices (DRBD) does synchronous replication of modified disk blocks. The advantage is consistently fast failover times. And because it is synchronous, incoming data is virtually guaranteed to be replicated, for maximum protection against data loss. The disadvantage is that traffic-handling volume may be decreased, due to the serial writes (replication must complete before the original data write can finish). This mechanism is also expensive to set up and maintain.
- Learn More: http://dev.mysql.com/doc/refman/5.5/en/ha-drbd.html
- MySQL Replication + DRDB + MySQL Heartbeat
- Replication and DRDB are not mutually exclusive. They can be used in combination with one another.
2.4.1 Comparing the Options
The following table compares those replication schemes:
Requirements Replication Replication + Heartbeat DRBD + Heartbeat Replication + DRBD + Heartbeat Availability Automated IP failover No Yes Yes Yes Automated database failover No No Yes Yes Typical failover time User/script-dependent Varies < 30 seconds < 30 seconds Automatic resynchronization of data No No Yes Yes Geographic redundancy support Yes Yes No Yes Scalability Supports Read-intensive applications Yes Yes No Yes Supports Write-intensive applications No No Yes Yes Maximum number of nodes per group One master, multiple slaves One master, multiple slaves One active (primary), one passive (secondary) node Maximum number of slaves Unlimited (reads only) Unlimited (reads only) One (failover only)
- Note: This table is based on information obtained from http://dev.mysql.com/doc/refman/5.5/en/ha-overview.html
2.4.2 MySQL Cluster Not Supported
MySQL Cluster has functionality limitations that prevent it from working with the platform. Some of the more significant limitations include:
- Limitation on number of objects in an instance (around 20,000)
- Temporary tables are not supported
- Limitations on Index size
- Savepoint and Rollback to Savepoint are not supported
- Replication is not supported
- Online schema changes are not supported
3 Platform Installation
3.1 Installation Files
3.1.1 Post-Installation File Hierarchy
When the platform is installed, these additional folders are created:
- docs
- Store the documents, attachments, and other files uploaded to the platform.
- tmp
- A work folder used for temporary files.
3.1.2 Configuring the MySQL Server
These instructions apply to the version of MySQL specified in the Software Requirements.
Note: MySQL configuration is a pre-requisite for the AgileApps installation.
Terminology:
- MySQL Server
- The database server which has MySQL Server software installed, configured, and running.
- MySQL Client
- Any other computer which connects to the database server and has the MySQL Client software installed, configured, and running.
- The client connects to the database server to access data, for example, webserver.
- Configuration file
- my.cnf, a MySQL configuration file that may be edited and used in the MySQL client or MySQL Server installation.
3.1.2.1 MySQL Settings
- Storage Engine
- default-storage-engine = innodb
- Server System Variables Configuration
- It is observed that READ_UNCOMMITTED does not work well with Row-Based Replication. Use one of the following configurations:
- a) Either STATEMENT-BASED or MIXED replication with the Transaction Isolation level dictated by business needs.
- b) ROW-BASED replication with transaction_isolation = READ-COMMITTED
- User Configuration
-
- Create a user account with password in MySQL
The Application Server will use the user details.
- The user account should have all MySQL privileges enabled on all databases
- Use the default root account that is created during the installation of MySQL. You can also create a non-root MySQL user to run Agile Apps. For more information, see Creating a non-root MySQL User to Run Agile Apps.
- Create a user account with password in MySQL
- sql-mode Configuration
-
- Add the following code to the my.cnf file located in the MySQL Client:
- sql-mode="PIPES_AS_CONCAT,ANSI_QUOTES"
- Add the following code to the my.cnf file located in the MySQL Server:
- sql-mode="PIPES_AS_CONCAT,ANSI_QUOTES"
- MySQL Configuration for UTF-8 Unicode character set for 5.7 Version
-
- Add the following code to the my.cnf file, located in the MySQL Client:
- default-character-set=utf8
- Add the following code to the my.cnf file, located in MySQL Server:
- character-set-server=utf8
- collation-server=utf8_general_ci
- The character set defines how records can be alphanumerically ordered (or grouped, sorted, filtered, indexed). The list of supported languages is determined by the character set available at the server level. Hence, the UTF-8 Unicode character set configuration is required as part of MySQL configuration.
- To specify character settings at MySQL configuration time:
For MySQL Version 5.7, see https://dev.mysql.com/doc/refman/5.7/en/charset-applications.html
- To specify character settings at MySQL configuration time:
- MySQL Configuration for UTF-8mb4 Unicode character set for MySQL 8.0 Version
-
Note: For MySQL configuration v5.7 to v8 upgrade, refer Upgrading to MySQL v8.
- Add the following code to the my.cnf file, located in the MySQL Client:
- default-character-set=utf8mb4
- Add the following code to the my.cnf file, located in MySQL Server:
- character-set-server=utf8mb4
- collation-server=utf8mb4_0900_ai_ci
- The character set defines how records can be alphanumerically ordered (or grouped, sorted, filtered, indexed). The list of supported languages is determined by the character set available at the server level. Hence, the UTF-8 Unicode character set configuration is required as part of MySQL configuration.
-
Note: If user does not specific in my.cnf or my.ini file then default MySQL 8 character set and collation are utf8mb4 and utf8mb4_0900_ai_ci taken by server.
- To specify character settings at MySQL configuration time:
For MySQL Version 8.0, see http://dev.mysql.com/doc/refman//8.0/en/charset-applications.html
- To specify character settings at MySQL configuration time:
- max_allowed_packet Configuration
- Add max_allowed_packet = 64M to my.cnf or my.ini
- For MySQL Version 5.7, see https://dev.mysql.com/doc/refman/5.7/en/charset-applications.html
- For MySQL Version 8.0, see https://dev.mysql.com/doc/refman/8.0/en/charset-applications.html
- Restart the MySQL server to implement the changes
- regexp_time_limit Configuration
The regular expression limit is expressed as the maximum permitted number of steps performed by the match engine, and thus affects execution time only indirectly. It is defined in milliseconds.
- Add regexp_time_limit=2048 to my.cnf or my.ini
- For MySQL Version 5.7, see https://dev.mysql.com/doc/refman/5.7/en/charset-applications.html
- For MySQL Version 8.0, see https://dev.mysql.com/doc/refman/8.0/en/charset-applications.html
- Restart the MySQL server to implement the changes
3.1.2.2 Populate the Timezone Tables
- The platform uses MySQL's timezone tables for timezone conversions. These tables are not automatically populated when MySQL is installed, so it necessary to do so after installation.
- Run the following program to initialize the MySQL timezone tables:
- mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql
- where /usr/share/zoneinfo is the standard Linux location for the time zone files. (Your system may differ.)
- For MySQL Version 5.7, see https://dev.mysql.com/doc/refman/5.7/en/mysql-tzinfo-to-sql.html
- For MySQL Version 8.0, see https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html
- mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql
3.1.3 MySQL Tuning Parameters
These section provides some typical MySQL configuration values that are appropriate for a production system. These values assume:
- 100Mb Primary Ethernet Card
- 32 GB Memory
- 2 Processors
- 2 Cores per Processor
- Switched Ethernet Port
- 64-bit Red Hat Enterprise Linux ES 4
Important:
- Before changing parameters, take the server and MySQL down.
Do not adjust parameters while MySQL is running. Doing so can potentially cause the database to be corrupted. - The settings shown here are an example. Consult the latest MySQL document for information on the best settings to use.
- Before changing parameters, take the server and MySQL down.
3.1.3.1 my.cnf / my.ini Options
interactive-timeout = 7200 wait-timeout = 7200 back_log = 200 max_connections = 350 max_connect_errors = 10 table_cache = 8192 max_allowed_packet = 48M binlog_cache_size = 8M max_heap_table_size = 120M sort_buffer_size = 2M join_buffer_size = 2M thread_cache = 200 thread_concurrency = 24 query_cache_size = 500M query_cache_limit = 1M ft_min_word_len = 4 thread_stack = 256K tmp_table_size = 120M
3.1.3.2 InnoDB-Specific Options
innodb_additional_mem_pool_size = 48M innodb_buffer_pool_size = 23G innodb_data_file_path = ibdata1:10M:autoextend innodb_file_io_threads = 4 innodb_thread_concurrency = 24 innodb_flush_log_at_trx_commit = 1 innodb_log_buffer_size = 8M odb_log_file_size = 256M innodb_log_files_in_group = 3 innodb_max_dirty_pages_pct = 90 #innodb_flush_method=O_DSYNC innodb_lock_wait_timeout = 300
Installing and Upgrading the LongJump Application Platform
3.2 License Requirements
About licenses:
- One platform license is required for each instance of the Application Server.
- A short term license is included in the initial installation.
- An email notification is sent out 30 days before a license is due to expire.
- A grace period extends for an additional seven days after the expiration date.
Daily notifications are sent during that period, until the license is renewed or the grace period ends. - To obtain a long-term license, contact your Software AG representative. You will receive an XML license file.
To install the new license:
- Copy the XML license file to {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN
- Restart tomcat
4 Platform Security
4.1 Managing SSL Certificates
4.1.1 Obtaining an SSL Certificate
The platform provides a default self-signed certificate which is used by the Application Server.
To obtain and install your own SSL Certificate, make a request to a Certificate Authority (CA). An SSL certificate authenticates a website to a web browser, part of a security protocol to manage secure data exchange.
The CA will accept your Certificate Signing Request and generate a certificate which identifies your website as a secured website.
To create a Certificate Signing Request (CSR)
- 1. Create a keystore and a private key:
cd {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN keytool -genkey -alias tomcat -keyalg RSA -keysize 2048 -keystore {keystore_filename}
- 2. Create a CSR from the keystore
keytool -certreq -keyalg RSA -alias tomcat -file certreq.csr -keystore {keystore_filename}
- 3. Submit the resulting file, certreq.csr, to the CA to obtain a certificate.
(When the certificate arrives, you are ready for the next step of steps.)
To Install the Certificate Obtained from the CA
Once you have obtained a certificate, you need to import it into the keystore.
But first, in addition to your certificate, the CA might provide a Chain/Root Certificate, which must also be imported. If you have received a chain certificate from the CA, then:
- 1. Copy the contents of the chain certificate into a file called chain
- 2. Import the chain certificate into your keystore:
keytool -import -alias root -keystore {keystore_filename} -trustcacerts -file chain
When the chain certificate (if any) has been imported, you are ready for the final step:
- 3. Import the certificate received from the CA:
keytool -import -alias tomcat -keystore {keystore_filename} -trustcacerts -file {certificate_filename}
-
Note: If you have SSL certificate and private key, follow the below steps:
- 1. Convert the private key and certificate to PKCS#12 format using OpenSSL. Assuming you have the private key file in .key format (private.key) and the certificate file in .crt format (VMNX-AALIND22.crt), use the following command:
openssl pkcs12 -export -inkey <private.key> -in <certificate.crt> -out <keystore.p12> -name <alias>
The default alias is set to 1.
- 2. Replace <alias> with the desired alias for the key entry.
- 3. Import the PKCS#12 file into the Java keystore using the keytool command:
keytool -importkeystore -srckeystore <keystore.p12> -srcstoretype PKCS12 -destkeystore <keystore.jks> -destalias <alias>
- 4. Replace <alias> with the alias used in the previous step.
5. Enter the appropriate passwords when prompted, including the source keystore password for the PKCS#12 file and the destination keystore password for the Java keystore.
- 6. Once you have successfully completed these steps, the certificate and private key should be imported into the Java keystore with the specified alias.
4.1.2 To update a Customer SSL Certificate in AgileApps
- 1. Stop the Application server.
- 2. Update keystoreFile and keystorePass values in âcom.softwareag.catalina.connector.https.pid-agileappsHttps-8284.propertiesâ file available under
- {install-dir}/profiles/IS_default/configuration/com.softwareag.platform.config.propsloader folder.
-
Note: The keystorePass value provided by you in plain text is encrypted automatically when you restart the AgileApps application server.
- 3. After updating the properties, place the certificate in the {install-dir}/profiles/IS_default/configuration/tomcat/conf folder.
- 4. Restart the memcached server and start the AgileApps application server.
4.1.3 Learn More
- Certificate Signing Request (CSR) Generation Instructions-Tomcat, at
https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&id=AR227
4.2 Controlling Port Access
4.2.1 Firewall Ports
Platform Application Servers are typically deployed behind a Firewall. The firewall needs to open the ports those servers use.
The default ports are:
- Non SSL port 80
- SSL port 443
The SSL port always needs to be open. If the application is to be accessed only using https, the http port can be blocked. If the platform's Sites capability will be used to provide a public URL, then the http port needs to be open.
-
Note: For secure communication, you should always access the platform using SSL (https://yourdomain/networking/Service). The platform provides a default self-signed certificate which is used by the Application Server. This certificate can be replaced with your own certificate, purchased from a certificate-signing authority. For more instructions on that process, see Managing SSL Certificates.
4.2.2 Changing Port Assignments
To change port assignments:
- Edit the following files available in {install-dir}/profiles/IS_default/configuration/com.softwareag.platform.config.propsloader as follows:
- com.softwareag.catalina.connector.http.pid-agileappsHttp-8283.properties - In this file, set the port value to 80 and redirectPort value to 443.
- com.softwareag.catalina.connector.https.pid-agileappsHttps-8284.properties - In this file, set the port value to 443.
Note: If you change the http port (80), you will need to specify the port number in the URLs for the document servers recorded in the database. To see those values:
Mysql> SELECT document_server,import_document_server,public_document_server FROM relationals.NETWORK_GLOBAL_PROPERTIES;
Those URLs are read when an application server starts, and used as the forwarding-target for document-access requests.
5 Backend Services
5.1 Managing Backend Services
The AgileApps Cloud platform has several backend services. Each of them can be run on a different server, to improve performance and scalability. They can be disabled, as well, if they are not used.
-
Important:
All backend services are enabled by default. But...- Backend services should not be enabled on a customer-facing web server, to avoid performance problems.
(The exception would be a single-user server used for development or proof of concept.) - With the exception of memcached, a backend service should not be enabled on more than one server, to avoid errors.
Immediately after installing a server, then, it is necessary to disable all backend services, except for those services that are intended to run on the current server--and then only if the current server is intended to be a backend server.
- Backend services should not be enabled on a customer-facing web server, to avoid performance problems.
5.1.1 About the Backend Services
An Application Server instance can run one or more of the following services. Enabling and disabling them determines which instance they run in--or whether they are available at all.
Service Default Status Description Report Scheduler Enabled Runs scheduled reports when they're due Import Enabled Imports of data into the database Export Enabled Exports data from the database Memcached Enabled Data caching mechanism (installed separately) Quartz Enabled Time keeper for all scheduled events.
5.1.2 Enabling and Disabling Backend Services
During installation, the networking.properties file is created. That file can be used to manually enable and disable backend services for an existing installation.
-
Important: Each background service should run on only one server.
- To view the server list
- Open {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN/networking.properties
- Servers are listed in this format: xxxxx_manager_instance=1
where "xxxxx" is one of "import", "report", etc.
- Servers are listed in this format: xxxxx_manager_instance=1
- To Disable a service
- Add a "#" character before the xxxxx_manager_instance=1 command
- # report_manager_instance=1
- Save the file
- Restart the Application Server
- To Enable a service
- Remove the "#" character from the xxxxx_manager_instance=1 command
- report_manager_instance=1
- Save the file
- Restart the Application Server
5.1.3 Configuring Backend Services
During installs and upgrades, it is only necessary to enable or disable a service on a platform instance. The install/upgrade process takes care of making the connections.
To change a configuration for an existing installation, changes need to be made manually, by adjusting the networking.properties file(s) and restarting the Application Server(s).
-
Note: A change in the configuration for either memcached, quartz, or the document service requires a restart of all Application Servers.
The general process is:
- Make the changes to the networking.properties files for all Application Servers.
- Restart the Application Servers.
These sections describe the changes to make:
Manually Configuring Backend Services
5.1.4 Configuring Document Services
-
Important: Changing the document service configuration requires a restart of all Application Servers.
Document uploads and downloads are handled by the document service. Managed documents include:
- Individual documents accessed through the documents tab
- Attachments to records
- File and image fields in records
The document service ensures that documents can be accessed from any application server. One of the App Servers should be designated as the document server. That server needs access to the storage device that will hold the documents.
-
Recommendation: A reliable storage device should be used for document storage--something on the order of a SAN or a RAID enabled disk array.
The URL of the document server can be configured by updating the following columns in the table relationals.NETWORK_GLOBAL_PROPERTIES:
- document_server - Stores documents that can be accessed only by authenticated users
- public_document_server â Stores public documents that can be accessed without logging into the platform
- import_document_server â Stores documents that are uploaded using data import
- Notes:
- The typical value for those properties is http://ip_address_of_document_server/networking
- Changing the document server location requires a restart of all platform instances.
-
Recommendation: All three values should be the same. In other words, exactly one server should be responsible for all of the document services.
5.1.5 Configuring the Quartz Scheduler
-
Important:
- Quartz must be enabled on at least one platform instance, or the platform won't run.
- It is recommended that you run Quartz on the back-end servers to avoid any performance issues in the front-end servers.
- If you have multiple instances, Quartz should run on only one of them, unless you follow the Quartz clustering instructions below.
- Changing the quartz configuration requires a restart of all Application Servers.
5.1.5.1 Enabling Quartz on a Platform Instance
- 1. Enable qrtz_manager_instance = 1. This is available at {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN/networking.properties
- 2. Edit {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN/AALSchedulers.xml
- 4. Add the following information:
- a. Database user name (an admin user)
- b. Password for that user
- Example:
<attribute name="org.quartz.dataSource.myDS.URL" value="jdbc:mysql:"/> <attribute name="org.quartz.dataSource.myDS.user" value=""/> -- root user <attribute name="org.quartz.dataSource.myDS.password" value=""/> -- password
-
Note: Do not change the value of org.quartz.scheduler.instanceName property.
5.1.5.2 Moving Quartz to Another Platform Instance
Suppose you have one application server instance, and you have decided to add another to handle the traffic volume. And let's say you also decide to move quartz to the new instance, to further reduce the load on the original server. To that, you disable Quartz on the first instance, and start it on the second.
- 1. Disable qrtz_manager_instance = 0. This is available {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN/networking.properties
- 2. For the second instance, follow the steps above, in Setting up Quartz on a Platform Instance
5.1.5.3 Setting up a Quartz Cluster
By default, Quartz is pre-configured to run as a cluster of instances running on different servers. To run successfully, however, each server must have the settings shown here. Otherwise, a Quartz instance will not know that the others exist, and will not lock job records that are in progress. Perform this configuration on any server you later add to the cluster.
- 1. Open {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN/AALSchedulers.xml
- 2. Check this line:
<attribute name="org.quartz.scheduler.instanceId" value="AUTO" [VERIFY THIS LINE]
- 3. Under # Configure JobStore, check these lines:
... # Configure JobStore <attribute name="org.quartz.jobStore.misfireThreshold" value="60000"/> [VERIFY THESE LINES] <attribute name="org.quartz.jobStore.isClustered" value="true"/> <attribute name="org.quartz.jobStore.clusterCheckinInterval" value="20000"/>
In addition, MySQL must be configured to set the appropriate transaction-isolation level and to specify row-based replication:
- Edit {MYSQL_INSTALLATION}/my.cnf
- Use the following settings:
transaction-isolation = READ-COMMITTED binlog-format = ROW
5.1.6 Configuring memcached
The MEMory CACHE Daemon is a high-performance, distributed-object caching system that minimizes user response time by caching application data and other elements required by programs running on the application server.
The platform uses memcached to store meta information which so that they can be obtained from the cache without accessing the database e.g. Object and field definitions, layout definitions, data policies etc. (Transactional customer data is not stored in memcached--that data is cached by the database.)
Data is stored in memcached using a lazy read mechanism. When some meta information is required, the platform checks to see if it is available in memcache. If it is available, the data is retrieved. If it is not available, a database read is performed, which in turn populates the entry in memcached. When the meta information changes due configuration changes by the user, the corresponding entry is flushed from memcached.
-
Important: The configuration for every instance must have the same memcached settings--otherwise, the caches won't be synchronized. Changing the memcached configuration therefore requires a restart of all Application Servers.
5.1.6.1 Managing memcached Servers
- To install memcached
- See http://memcached.org/ for installation instructions.
- To start memcached
-
- Execute the memcached script located in the /bin directory:
./memcached -d start -p 11211 -u root -m 25
- where:
- -p is the port number (11211)
- -u is the user (root)
- -m is the amount of memory in MB (25)
- (Change the parameters to the values used for your installation.)
- where:
- To stop memcached
./memcached -d stop</tt>
5.1.6.2 Configuring Application Servers to Use memcached
- To enable memcached
- Edit networking.properties
- Remove the leading # character from the line
#MemCachedServers=localhost:11211
- To change the default port
- The default port is 11211. To change the port assignment, modify the value(s) for MemCachedServers in networking.properties.
- To configure multiple memcached instances
- Multiple memcached instances are recommended, so that cache access is distributed. Do that by specifying the IP address and port for each memcache server in the networking.properties file.
-
Important: If there are multiple memcached servers, then:
- Every instance should have an identical list of servers.
- The servers should be in the same order in each list.
- If the configuration changes, all application servers should be restarted.
- Learn more: Adding Additional memcached Servers
- To bring up the servers
- After making the changes to the networking.properties files for all Application Servers, (re)start servers in the required Server Restart Sequence.
6 Performance Optimizations
Configuring MySQL and LongJump to Run on Separate Servers Configuring Apache to serve static content from LongJump Adding Additional LongJump Servers
6.1 Adding Additional memcached Servers
Adding an additional memcached servers helps to distribute the cache and balance the load across the memcached servers.
- To add an additional memcached server
- Edit networking.properties on all instances.
- Update the MemCachedServers entry
- Restart all servers, following the Server Restart Sequence
Note: The sequence is important!
- Example
There is one memcached server running on serverA (192.168.0.1) on port 11211. On each instance, the networking.properties has this:
MemCachedServers=192.168.0.1:11211
A second memcached server is brought up on serverB (192.168.0.2) on port 11211. The networking.properties file is edited on all instances so it looks like this:
MemCachedServers=192.168.0.1:11211 192.168.0.2:11211
After following the Server Restart Sequence, all instances are using the new, distributed cache.
If a 3rd memcached server is added (192.168.0.3), the networking.properties file will look like this:
MemCachedServers=192.168.0.1:11211 192.168.0.2:11211 192.168.0.3:11211
Again, the Server Restart Sequence is followed to enable the new configuration.
7 Application Server Administration
7.1 Server Restart Sequence
When an installation employs memcached or the Messaging Server, it is important to follow this sequence when restarting servers:
# STOP THE MESSAGING SERVER, if one is running: /etc/init.d/messaging stop {hornetq-folder}/bin/stop.sh # STOP ALL APPLICATION SERVERS # On each server: {install-dir}/profiles/IS_default/bin/shutdown.sh # STOP ALL memcached SERVERS # On each server: /bin/memcached -d stop # START ALL memcached SERVERS # On each server: /bin/memcached -d start -p {port} -u {user} -m {MB_of_memory} # Typical values: # Port: 11211, User: root, MB of memory: 25 # START ALL APPLICATION SERVERS # On each server: {install-dir}/profiles/IS_default/bin/startup.sh # START THE MESSAGING SERVER, if you're running one: {hornetq-folder}/bin/start.sh /etc/init.d/messaging start
- Considerations
-
- Stopping application servers ensures that they aren't adding entries to the cache.
- Stopping memcached makes sure that the cache is flushed.
- Those two steps can occur in either order. It is the next two for which order is critical:
- Restarting memcached first makes sure that a clean copy of the cache is available.
- When the application servers come up, they use the clean cache.
7.2 Start the Application Server
- Login as root
- {install-dir}/profiles/IS_default/bin/startup.sh
7.3 Restart the Application Server
- Login as root
- {install-dir}/profiles/IS_default/bin/restart.sh
7.4 Stop the Application Server
To stop the application server, you kill the Apache Tomcat container it's running in.
- Login in as root
- {install-dir}/profiles/IS_default/bin/shutdown.sh
7.5 Accessing the Application Server
After the application server has been started:
- 1. Access the Service Provider URL: http://{yourDomain}/networking/Service?t=1&targetpage=ViewPort.jsp:
- Where:
- {yourDomain} is the platform domain. For example: yourcompany.com
- Default username and password: Use the credentials you typed into the Software AG installer.
-
Note: These credentials are only applicable to the on-premise fresh installations of AgileApps and not for AgileApps cloud.
- The platform will request that you change your password after you login the first time.
- 2. Setup the Service Configuration with these required parameters:
- Configure the Service Settings and specify Service and Domain names:
Parameter Description Typical Value Service Name Name of the service provider Financiocorp Services Prefix for Service Domain Optional subdomain name Allowed characters: a-z, A-Z, 0-9, - (alphanumeric, plus hyphen)
Example: service
Service Domain The Domain Name part of the URL mydomain.com Domain URL Read Only
Automatically populated as:- Prefix + Service Domain
service.mydomain.com
7.6 Creating a Tenant
- After Login, create a new tenant from a web browser using the URL as follows:
- https://yourdomain.com/networking/Service?t=2308
- Where yourdomain.com is the name of your domain
8 Database Administration
Recognizing the LongJump Databases
8.1 Database Administration Tips
- Ensure that all database servers and web servers are configured to be in the same timezone, regardless of where they are geographically located.
- You should have different user accounts with different access permissions. These are typical:
- A root user with all permissions, accessed from a specific host only.
- For access from other web servers, one root user for each host.
- An admin user with limited privileges to do a DB dump, monitor the DB server, and/or monitor the replication server.
- A read only user who cannot modify the database for reports, backups, and exports.
- A replication user that is used only by the replication process.
- Always do DB dumps, reporting, and exports from the replication server, as those activities introduce locks that interfere with users.
- Analyze queries from the replication server. Do not login as root.
- Do not login as root on the production server unless executing a DML or DDL.
- Always use transactions when executing a DML (begin- and end-rollback).
8.2 Database Backup and Recovery
MySQL can be backed up using the mysqldump command - http://dev.mysql.com/doc/refman/5.5/en/mysqldump.html
Tip: The replicated database server should be used for backups.
To set up for it, see Configuring MySQL to Run on a Separate Server
8.2.1 Standard Database Backup
- Dump the database
mysqldump âuroot âpxxx âall-databases âquick âroutines âresult-file=dumpfile.sql
- Dump the database at regular intervals, using a Linux cron job
-
- Put the mysqldump command in a shell script - say, xyz.sh
- Setup cron job e.g. every day at 6 pm
0 18 * * * /yourscriptlocation/xyz.sh > /somedirectory/xyz.out 2>&1
- Use tar to compress the resulting dump file, to save space.
- Maintain daily backups for ten days or so, to reclaim the space they use.
- Maintain monthly backup. Save the dump of last day of the month, for example, and retain the dump file for a year or so.
- Store long-term backups offsite.
8.2.2 Standard Database Restore
- Restoring all data from a dump file
- Drop all the databases.
- Restart mysqld
- Start the restore process and run in background:
mysql âuroot âpxxxx < dumpfile.sql > dumpfile.out 2>&1 &
- Restoring a single database or table from a dump file
- Drop the old database/table:
mysql âuroot âpxxxx $database < dumpfile.sql > dumpfile.out 2>&1 &
8.2.3 Learn More
For more detailed information on database backup and recovery in MySQL, see:
Managing Email Credits with Vertical Response
9 Housekeeping
9.1 Regular Maintenance Ensures Top Performance
Regular housekeeping helps to keep the platform running at peak efficiency.
You'll know that housekeeping is necessary when your Platform Monitoring reveals a decreased level of performance, despite your Performance Optimizations and Platform Tuning.
But regular, planned maintenance can prevent such slowdowns from occurring in the first place.
As part of that process, you will be:
9.2 Purging and Archiving Data on the Platform
To keep storage requirements to a minimum, perform these actions at regular intervals, using the platform GUI:
- Audit logs are purged after a retention period which can be configured.
- Debug logs used for development are retained for a day.
- When a record is deleted, it is moved to the Recycle Bin where it is purged after 30 days.
- The platform provides a data export mechanism which can be used for data archiving.
- Data can also be read using the REST API and saved elsewhere.
9.3 Cleaning Up Temporary Files on the Server
Every few months, it's a good idea to remove old temporary files, to keep them from accumulating on the server. To do that:
- Delete old CSV files in the documents folder.
For example, delete *.csv files that are more than 5 days old. - Delete old files in the temp folder.
For example, delete files more than 5 days old.- To find the location of the temp folder:
- From the Service Provider URL: http://{yourDomain}/networking/Service?t=1&targetpage=ViewPort.jsp
- Go to Settings > Service Provider Settings > Service Configuration.
- To find the location of the temp folder:
- Delete old tomcat logs.
For example, delete files more than 10 days old in {install-dir}/profiles/IS_default/logs - Delete old apache logs.
For example, delete files older than 10 days in {apache-install-dir}/apache/logs
10 Platform Monitoring
10.1 Heartbeat Check
The Heartbeat Check checks to make sure that the platform is still running. It's accomplished by visiting a URL which gets processed by Tomcat. (This lightweight URL gets an gets an immediate response from Tomcat, if it is running.)
-
Note: If Tomcat runs out of memory is not responding for some other reason, you won't be able to tell by monitoring the port or the process. The Heartbeat Check is the only way to know.
If you happen to be logged in, you can visit https://{yourDomain}/networking/rest in your browser. If Tomcat is running, you get a list of REST APIs supported by the platform.
To run the same test without logging in, use a REST client to visit the URL and specify the HEAD method (rather than GET, PUT, or POST). If Tomcat is running, you get an http return code of 200.
10.2 Monitoring Servers
Any server in the system can become a bottleneck, at some point, so it's a good idea to continually monitor the health of the critical processes that run on them. This section lists the components to monitor on each server.
10.2.1 Monitoring Application Servers
- tomcat availability and CPU utilization. Check threads, connection pool size, sticky sessions, KeepAliveRequests, etc.
- GC â Allocation and de-allocation of memory on the JVM. (Monitoring and Tuning Garbage Collection)
- OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics)
10.2.2 Monitoring Web Servers
- apache-httpd availability
- OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics)
10.2.3 Monitoring Database Servers
- mysql availability. Check caches, buffers, connections, timeouts. (Monitoring and Tuning MySQL)
- OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics)
- Note: If replication is employed, scripts can be written to check the health of the replication and report replication lag times.
10.2.4 Monitoring memcached Servers
- memcached availability
- OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics)
10.2.5 Monitoring the Network
- Dropped packets
- Socket wear and tear
10.3 Monitoring Processes
These pages tell you how to monitor the critical processes on each server:
10.3.1 Monitoring and Tuning Garbage Collection
To monitor the frequency of Garbage Collection inside the JVM:
$java_home/bin/jstat -gcutil pid <intervalmillis> [root@web1]# /usr/local/java/bin/jstat -gcutil 12929 5000 S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 42.66 37.32 12.82 26.96 4845 145.414 26 1.745 147.159 0.00 42.66 37.78 12.82 26.96 4845 145.414 26 1.745 147.159 0.00 42.66 57.70 12.82 26.96 4845 145.414 26 1.745 147.159
Tuning parameters are set in catalina.sh under CATALINA_OPTS
Learn More: http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html
10.3.2 Monitoring and Tuning MySQL
- Running on a 64-bit machine, there should be plenty of memory.
- 20G of memory or more should be allocated to MySQL (highly recommended).
- Allocate a good amount of memory to innodb_buffer_pool_size - the combined cached for data and the index.
- TimeOuts
- interactive_timeout, wait_timeout,innodb_lock_wait_timeout.
- Number of file descriptors (number of files that MySQL can have open)
- Open_files_limit.
- Also: Check that you have a sufficient number of file descriptors available in the OS.
- To monitor the number of active threads, live threads, and open threads:
- show processlist or show full processlist
- After modifying tuning parameters in my.cnf, always restart mysql.
Learn More: http://dev.mysql.com/doc/refman/5.5/en/server-parameters.html
10.3.3 Monitoring OS Statistics
Before tuning any stack, whether for a web server or a database server, you should know how heavily system resources are being used. To get the information, use the following commands:
- top
top - 15:50:31 up 309 days, 21:10, 1 user, load average: 0.06, 0.16, 0.15 Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12929 root 18 0 3680m 1.2g 12m S 17.6 32.3 76:19.02 java
- mpstat
03:51:30 PM CPU %user %nice %system %iowait %irq %soft %idle intr/s 03:51:30 PM all 1.79 0.00 0.29 0.43 0.01 0.06 97.42 425.80 03:51:30 PM 0 3.22 0.00 0.43 0.47 0.02 0.21 95.65 151.15 03:51:30 PM 1 1.08 0.00 0.21 0.68 0.00 0.01 98.02 103.64 03:51:30 PM 2 1.65 0.00 0.28 0.38 0.00 0.02 97.68 96.45 03:51:30 PM 3 1.24 0.00 0.23 0.19 0.00 0.01 98.33 74.56
- vmstat
procs -----------memory----------- --swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 22264 1110200 168460 1120660 0 0 1 39 0 0 2 0 97 0
- free âm
total used free shared buffers cached Mem: 3945 2862 1083 0 164 1094 -/+ buffers/cache: 1603 2342 Swap: 1027 21 1005
Learn More:'
10.4 Monitoring Ports
Experience has shown that monitoring the processes and doing a health check tends to be a lot more useful than monitoring ports. But if you need to, here are the ports to monitor:
- Apache-httpd - tcp/80 and tcp/443
- memcached - tcp/11211
- MySQL - tcp/3306
- Tomcat - The ports Tomcat is listening on are listed in the following files available under {install-dir}/profiles/IS_default/configuration/com.softwareag.platform.config.propsloader folder:
- com.softwareag.catalina.connector.http.pid-agileappsHttp-8283.properties
- com.softwareag.catalina.connector.https.pid-agileappsHttps-8284.properties
10.5 Monitoring Memcached
Performance statistics can be retrieved using memcached commands.
Learn More: http://code.google.com/p/memcached/wiki/NewCommands#Statistics
11 Platform Tuning
11.1 Tuning Tomcat
The platform is a web application that runs in an Apache Tomcat container. The parameters listed in this section can be modified to optimize Tomcat's performance.
11.1.1 Connector Port Parameters
- Thread Allocation
- maxThreads, minSpareThreads, maxSpareThreads, maxkeepAliveRequests
- Timeouts
- connectionTimeOut, keepAliveTimeOut, connetionLinger
11.1.2 Connection Pooling Parameters
- initialSize, maxActive, maxIdle, maxWait, removeAbandoned, removeAbandonedTimeout
11.1.3 Learn More
11.2 Tuning Report Threads
The number of report threads can be configured in the networking.properties file:
- report_threads=2
- Assigns number of threads to be used by the report server.
Generally, two threads devoted to reporting is about right. If reports aren't arriving on time, consider incrementally expanding the number of threads to see if performance improves (while keeping an overall eye on system resources, to ensure that increasing the number of threads does't incur other bottlenecks.
For maximum reporting efficiency (if needed), setup a dedicated backend server for reporting. Learn More: Managing Backend Services
12 Troubleshooting
12.1 Troubleshooting Quartz
If quartz stops processing jobs from the quartz queue, all scheduled jobs are listed in the NETWORK_SCHEDULE_POLICIES table. Ideally, there should be only 100 or so records/jobs scheduled in NETWORK_SCHEDULE_POLICIES.
Quartz lop entries can be viewed in relationals.log on the server where quartz is enabled. The log entries will look like this, where the worker threads are the ones that pick up and process jobs in the queue:
<timestamp> MasterScheduler ... <timestamp> [PlatForm_Worker-1] ... <timestamp> [PlatForm_Worker-2] ... <timestamp> RelationalsSchedulerWorker-1 ... <timestamp> RelationalsSchedulerWorker-2 ... ...
Here are some possible ways that quartz could get stuck:
- There are too many jobs/records in NETWORK_SCHEDULE_POLICIES
- Numbers in the 1,000's may cause quartz to become bogged down. You might want to determine what caused these many jobs to be inserted in the queue. If it is safe to delete them, they can be removed from the queue. Or they could be copied to a temporary table re-inserted later, at a time when there is no load on the server.
- Quartz might run into an error that causes the quartz thread to die.
- In that case, restarting the application server will bring the quartz thread back to life.
-
Tip: Set up a cron job that gets count of entries in the NETWORK_SCHEDULE_POLICIES. If the count hasn't changed after 20 mins or so, it usually means quartz is not picking up the jobs. (Alternatively use the REST APIs to create a task that monitors the status of that table.)