PowerPoint-presentatie

January 6, 2018 | Author: Anonymous | Category: Engineering & Technology, Computer Science, Databases
Share Embed Donate


Short Description

Download PowerPoint-presentatie...

Description

How to get your hands on the data of your LMS (e.g. Blackboard) Alan Berg Co-chair SIG SURF / Community officer Apereo LAI

Acknowledgements

Resources mentioned in Resources.txt, including: Understanding The Advanced Reporting Database John Knight 2005: – 5 relevant slides are copied from this presentation (orange background). Clip art: https://openclipart.org/

WARNING!

A lot of information will be covered in a short time.

Agenda

Who am I Motivation BlackBoard Processes BlackBoard database schema Pulling data out of BlackBoard Simple Analysis in R Brain Storm over possibilities

Wrangler | 07 maand 2014

4

Who am 

  

 





I have relevant practical experience. Developed building Blocks Integrated stuff with UVA's SIS Run Quality Assurance on a number of large scale projects such as UvA's Scale out Reported bugs Presented a couple of times (a long time ago) at BlackBoard conferences Old, old stuff: Blackboard 6 usage patterns and implications for the Universiteit van Amsterdam: http://www.uva.nl/binaries/content/documents/personalpages /b/e/a.m.berg/en/tab-three/tab-three/cpitem%5B41%5D/asset Pulled out data from BlackBoard for research projects

Motivation To break down walls that get in the way of getting at your data. 



By providing researchers with the details needed to communicate with the people that hold the keys to the data. By providing system /functional administrators an idea of the processes needed by researchers to use the data

Questions for the audience

  

1

Point to a Google doc A quick count of the audience roles A quick question to the audience of what they see as barriers to their research work.

BlackBoard Processes Motivation: Being specific with processes, filenames and locations so that it is easier to find and do stuff

Take homes





 

There is a set of best practices that should be followed BlackBoard files that matter  Bb-config.properties  Bb-tasks.xml Scheduled tasks for reporting The process of moving tables to a backup database

Best Practices Accessing the Database 



 

http://www.edugarage.com/display/BBDN/Blackboard+Open+ Database+-+OpenDB  Protect your data  Work with copies of your data  Remember to test performance  Use tools to access your databases Don't reinvent the wheel  See whats out in the wild  Re-use and improve artefact's Don't be arrogant: Ask the team that administers the system Look at the Open Database Schema

Questions for the audience

 

2

Point to a Google doc Additional best practices for accessing the Database

BB-config.properties 

 





Global configuration file for Blackboard. Options include:  Database  JVM tuning  Content Management  Server threads  Mail servers Performance depends on tuning this file Named a property file. Follows a specific text format:  bbconfig.unix.httpd.ssl.portnumber=443 Location similar to:  /app/blackboard/config/bb-config.properties Applying changes via a script PushConfigUpdates.bat/.sh

BlackBoard Processes Code that can run on any BlackBoard Server to get facts SECURE YOUR DATASOURCE TEST AGAINST A COPY OF YOUR DB 1) 2) 3) 4)

Create a user that can read BB-config.properties Create code that reads BB-config.properties Run Code through a scheduled task Generate time stamped reoprts or Text dumps

5)

LATER Improve code based on requirements

PurgeAccumulator job  

Default runs at 1 AM Does three things  Summarizes data  Moves data to a backup DB  Purges Production pf data older than a certain date  Configured via bb-tasks.xml  Runs stored procedures in the database

The gruesome details

Schema Description

Number of supporting records are increasing per version

2012 – Partial list 2007 – Full List

Titel van de presentatie | 00 maand 0000

Questions for the audience

 

3

Point to a Google doc Based on the 2007 table set which sort of questions can you ask when combined with SIS? – Promise to look at sending the relevant SQL statements or publishing to the LASI Utrecht website

Building Blocks - More   

  







Provides supporting services And JSP (GUI) libraries From experience, permissions structure changes between versions. Subtle changes in services Database schema is actually more stable. SQL queries more raw but are more efficient than service calls Was easier to learn Building blocks by code completion in IDE (E.G: Eclipse) than through the documentation Steep learning curve  http://workgroups.clemson.edu/DCIT2803_BBDEV/Tut orial/BBTutorial.pdf Take home: If you just want the data go directly to the

Bb-manifest.xml and Code Don't do this without a safety net ConnectionManager conman; int i=0; private Connection bbConnectDatabase() throws ConnectionNotAvailableException, InterruptedException { BbDatabase bbDb = DbUtil.safeGetBbDatabase(); conman = bbDb.getConnectionManager(); while(conn == null && iconnect($config->param("ConnectionSettings.dsn"), $config->param("ConnectionSettings.dbuser"), $config->param("ConnectionSettings.pass")) or die "Connection Error: $DBI::errstr\n"; generate_user_report(); $dbh->disconnect

;

Perl take home  

  

  

Easy to connect Just different configuration for queries to the archive database than production or development Perl is a well known language Perl is EXCELLENT for text processing Perl can generate unreadable code and not really a team language (personal opinion) Great for writing functionality quickly Excellent for manipulating and merging data Understood by many System administrators

Favorite tool – SQL Developer     



Everyone has their favorite tools Can export data efficiently Practice SQL statements Known to a wide community What are the ethics and security implications

STOP and ask Hendrik L to reinforce message from his presentation

Questions for the audience

 

4

Point to a Google doc Have you any favorite tools for use with databases or combining data sources?

Titel van de presentatie | 00 maand 0000

ASR Schema Elements

Table Name(s)

Data Stored

ACTIVITY_ACCUMULATO R SYSTEM_TRACKING

End user events.

APPLICATION COURSE_MAIN COURSE_ROLES COURSE_USERS DATA_SOURCE INSTITUTION_ROLES NAVIGATION_ITEM SYSTEM_ROLES,USERS USER_ROLES

Daily summary stats generated by summarization task in PurgeAccumulator process Stores supporting data referenced by ACTIVITY_ACCUMULATOR for ASR Reporting

Data Rentention in main schema Up to 360 days of data No limit (never purged) No limit. Tables are never purged automatically.

Data Structure

Activity Accumulator Dictionary

Column

Description

PK1 EVENT_TYPE

Primary Key Type of Event Posted

USER_PK1

Relates to USERS.PK1

COURSE_PK1

Relates to COURSE_MAIN.PK1

GROUP_PK1

Relates to GROUP.PK1

FORUM_PK1

Relates to FORUM_MAIN.PK1

INTERNAL_HAND LE CONTENT_PK1

Relates to NAVIGATION_ITEM.INTERNAL_HANDLE Relates to COURSE_CONTENTS.PK1

Activity Accumulator Dictionary

Column

Description

DATA

Additional information dependant on event type.

TIMESTAMP STATUS

Date/Time the event was posted 1 unless error occurred

MESSAGES

Error message details

SESSION_ID

Relationship to SESSIONS.PK1

REMOVE DATA and MESSAGES as might contain personal information?

Activity Accumulator Events

Event

Description

Relationships

LOGIN_ATTEMPT

A user attempted to login to the system. Captures the session id of the attempted login. A user clicked the logout button

SESSION_ID

LOGOUT

SESSION_INT PAGE_ACCESS

A user connected to the system and a session was created A page was accessed

COURSE_ACCES S

A page within a course was accessed

SESSION_ID USER_PK1 SESSION_ID SESSION_ID USER_PK1 NAVIGATION_ITEM SESSION_ID USER_PK1 COURSE_PK1 NAVIGATION_ITEM

Data Structure   









PK1 are keys to other tables PK1 are effectively random numbers As long as we do not combine with other tables then pseudo anonymous. Only combine when we have opted in Data and messages columns may contain personal information. Question for Hendrik L: How anonymous do we have to be? 5] Question for audience: How secure do we have to be with the data and how long can it live for? 6] Question for the audience: Do we need an audit trail?

Titel van de presentatie | 00 maand 0000

Example: Export in SQL developer

SQL Examples - General select event_type, count(*) from BackupDB_name.activity_accumulator group by event_type EVENT

2007

2012

LOGOUT

781618

1174147

SESSION_INT

20875358

20241290

CONTENT_ACCESS

5284051

14205997

COURSE_ACCESS

42266577

57808666

MODULE_ACCESS

15199704

66422352

LOGIN_ATTEMPT

3609998

PAGE_ACCESS

1528547

82827763

TAB_ACCESS

9715716

101974

7430655

eventTypes2007
View more...

Comments

Copyright � 2017 NANOPDF Inc.
SUPPORT NANOPDF