Saturday, March 4, 2017

Descriptive statistics in Python, caseStudy(Passenger arrival data analysis at metro stations) - P1.

Before the one goes with statistics, its important to know the difference between descriptive statistics and statistical inference. Descriptive statistics care about the procedures (for) the collection, summarization, and usually the presentation of quantitative data so that we can describe the relationship among the problem variables.

Statistical inference procedures are used to (make decisions) on the basis of (computed measures) taken from (samples). How both descriptive and inferential statistics correlate is simple; the computed  measures used in inference process are in fact descriptive measures (e.g. average, variance), easy so far, isn't it?

Lets layout another point that shows the difference between the the two methods, now its time to talk about the workflow of the statistical data analysis process, in more simpler words, how statisticians and data analysts tackle any given problem? well, Assume that we are data analysts working for Cairo metro, and we have been requested to check if the number of trains scheduled to run every hour are sufficient to meet the expected passenger demand with the highest possible rates of passenger comfort or not? we lay-out this problem in step by step approach and we are going to use our skills in statistical data analysis to do the task. but how shall we start and what steps shall we go through?

The answer is depicted in the following diagram:

Now lets proceed with the above mentioned method to analyze such data, the target of the current phase is just to understand some basic characteristics of arrival behavior:

Gathering Data:

For our case study, we gather the data of passenger arrival at every metro line station, thanks for smart ticketing systems and metro gates applied in most of the metro networks to avail such data for us. Assume the data is stored into a database tables where every table stores a one day passenger arrival data. and the passenger arrivals are summed per minute not by individual passenger arrival event. The table columns are the line stations and there is a row for every minute.

Wednesday, September 9, 2009

Introduction to software architecture

1. The need for software architecture:

· Old computer system problems depended on algorithms and data structures to solve them.
· As the size and complexity of software systems increases, the design problem goes beyond the algorithms and data structures of the computation.
· Designing and specifying the overall system structure emerges as a new kind of problems.
· System Structural issues include
a. Gross organization and global control structure. (Organization: System organizations).
b. Protocols for communication.
c. Synchronization.
d. Data access.
e. Assignment of functionality to design elements.
f. Physical distribution.
g. Composition of design elements.
h. Scaling and performance.
i. Selection among design alternatives.
· Software architecture is the process of defining a structured solution that meets all of the technical and operational requirements, while optimizing common quality attributes such as performance, security, and manageability
· Software architecture seeks to build a bridge between business requirements and technical requirements by understanding use cases, and then finding ways to implement those use cases in the software
· Software architecture discipline is centered on the idea of reducing complexity through abstraction and separation of concerns.
· There is no industry-standard, universally-accepted definition of Software Architecture
· As a maturing discipline with no clear rules on the right way to build a system, designing software architectures is still a mix of art and science.
· Modern thinking on architecture assumes that your design will evolve over time and that you cannot know everything you need to know up front in order to fully architect your system.
· How a system will support key business drivers is described via scenario as non- functional requirements of a system, also known of quality attributes, determine how the system will behave.
· Every system is unique due to the nature of the business drivers that supports, as such the degree of the quality attributes exhibited by a system such as:
o Fault tolerance *1
o Backward compatibility
o Extensibility
o Reliability
o Maintainability
o Security
o Usability
o Availability
· Understand your requirements and deployment scenarios first so that you know which quality attributes are important for your design
· Software architecture also described as the strategic design.
· Software architecture design is not entirely different from existing software design methodologies. Rather it complements them with additional views of a system that have not been traditionally handled by methodologies like OO design.

Strategic design:
Is an activity concerned with the global design constraints such as:
i. Programming paradigm
ii. Architectural Style
iii. Component based software engineering.
iv. Design principles
v. Low governed regularities

Detailed Design (Tactical design)
Is an activity concerned with local design constraints such as:
i. Design patterns
ii. Architectural Patterns
iii. Programming idioms
iv. Refactoring
Fault Tolerance:
The ability of a system to respond gracefully to an unexpected hardware or software failure. There are many levels of fault tolerance, the lowest being the ability to continue operation in the event of a power failure. Many fault-tolerant computer systems mirror all operations -- that is, every operation is performed on two or more duplicate systems, so if one fails the other can take over.
Keep in mind that the architecture should:
- Expose the structure of the system but hide the implementation details.
- Realize all of the use-case scenarios.
- Try to address the concerns of various stakeholders.
- Handle both functional and quality requirements.
Intension / Locality hypothesis
According to intension / locality hypothesis the distinction between strategic and tactical design is defined by the locality criterion, according to which a statement about software design is non-local if and only if:
The program that satisfies it can be expanded into a program which does not.
Ex. The client server architectural is strategic because a program that is built by this principle can be expanded into a program which is not client server (like adding peer to peer nodes).

2. Architects Framework:
1. Components:
Encapsulate some coherent set of functionality
2. Connectors:
a. Represent interactions as varied as procedure call, event broadcast, database queries, and pipes.
b. Realize the runtime interaction between components
3. Constraints
Ex. Topological (No cycles for example)

The architecture meta-frame
3. SW Architecture in SW process
- The architecture of software system can be specified in a document called the architectural description.
- Enterprise systems architectures:
o Software Architecture.
o Business Architecture.
o IT Architecture.
o Data Architecture.

Views about software life cycle
- Management view
- Software Engineering view
- Engineering design view
- Architectural View

Management View
Inception Phase
Elaboration Phase
Construction Phase
Deployment and Transition

LCO: Life cycle objective milestone.
LCA: Life cycle architecture mile stone.
IOC: Initial operational capability mile stone.

Deployment and Maintainance
Requirement Analysis and Specification
Design Phase
Implementation and TestingSoftware Engineering View

Product PlanningEngineering Design View
Conceptual Design
Endpoint Design
Detail Design

Pre-design PhaseArchitecting View
Domain analysis phase (Requirements)
Schematic Design Phase
Design development Phase
Building phase

4. SW architectural styles and patterns
We can distinguish among architecture styles by simply answer the following questions:
1. What is the structural pattern –Components, connectors, and constraints?
2. What is the underlying computational model?
3. What are the essential invariants of the style?
4. What are some common examples of its use?
5. What are the advantages and disadvantages of using that style?
6. What are some of the common specializations?
Some famous architectural styles Categories:
· Data Flow Architecture Batch Sequential Pipe & Filter Architecture Process-Control Architecture
· Data Centered Software Architecture Repository Architecture Style Blackboard Architecture Style
· Hierarchy Architecture Main/Subroutine Software Architecture Master/Slaves Software Architecture Layered Architecture Virtual Machine
· Implicit Asynchronous Communication Software Architecture Non-Buffered Event-Based Implicit Invocations Buffered Message-Based Software Architecture
· Interaction Oriented Software Architecture Model-View-Controller Presentation-Abstraction-Control (PAC) Architecture
· Distributed Architecture Client/ Server Multi-tier
Service based Multi-tier
Service-Oriented Architecture (SOA)

5. Cross-Cutting Concerns
Cross-cutting concerns represent key areas of your design that are not related to a specific layer in your application.

The following list describes the key cross-cutting concerns that you must consider when architecting your applications:
· Authentication. Determine how to authenticate your users and pass authenticated identities across the layers.
· Authorization. Ensure proper authorization with appropriate granularity within each layer, and across trust boundaries.
· Caching. Identify what should be cached, and where to cache, to improve your application’s performance and responsiveness. Ensure that you consider Web farm and application farm issues when designing caching.
· Communication. Choose appropriate protocols, reduce calls across the network, and protect sensitive data passing over the network.
· Exception management. Catch exceptions at the boundaries. Do not reveal sensitive information to end users.
· Instrumentation and logging. Instrument all of the business- and system-critical events, and log sufficient details to recreate events in your system. Do not log sensitive information.

6. Baseline and Candidate Architectures
Baseline architecture describes the existing system—it is how your system looks today. If this is a new architecture, your initial baseline is the first high-level architectural design from which candidate architectures will be built.

Candidate architecture includes the application type, the deployment architecture, the architectural style, technology choices, quality attributes, and cross-cutting concerns.

7. Key Architecture Principles
1. Do not attempt to over-engineer the architecture.
2. Your design will generally need to evolve during the implementation stages of the application as you learn more, and as you test the design against real-world requirements. Create your architecture with this evolution in mind so that it will be agile in terms of adapting to requirements that are not fully known at the start of the design process.
3. Build to change over build to last. Wherever possible, design your application so that it can change over time to address new requirements and challenges.
4. Model to analyze and reduce risk. Use threat models to understand risks and vulnerabilities. Use design tools and modeling systems such as Unified Modeling Language (UML) where appropriate.
5. Models and views are a communication and collaboration tool. Efficient communication of design principles and design changes is critical to good architecture. Use models and other visualizations to communicate your design efficiently and to enable rapid communication of changes to the design.
6. Identify key engineering decisions. Use an architecture frame to understand the key engineering decisions and the areas where mistakes are most often made. Invest in getting these key decisions right the first time so that the design is more flexible and less likely to be broken by changes.
7. Consider using an incremental and iterative approach to refining your architecture. Do not try to get it all right the first time—design just as much as you can in order to start testing the design against requirements and assumptions. Iteratively add details to the design over multiple passes to make sure that you get the big decisions right first, and then focus on the A common pitfall is to dive into the details too quickly and get the big decisions wrong by making incorrect assumptions, or by failing to evaluate your architecture effectively.
8. Use architecture evaluation to determine the feasibility of your baseline and candidate architectures. Consider the following techniques for architecture evaluation:
• Architecturally significant use cases. Test your design against use cases that are important to the success of your application, and which exercise a significant portion of the design.• Scenario-based evaluations. Use scenarios to analyze your design with a focus on quality attributes. Examples of scenario-based evaluations are: Architecture Trade-off Analysis Method (ATAM), Software Architecture Analysis Method (SAAM), and Active Reviews for Intermediate D

Wednesday, November 5, 2008

Why using workflows

  1. Workflows can handle long running work by persisting to a durable store, such as a database, when idle and loading again once there is work to do.
  2. Workflows are a declarative way of writing programs by linking together pre-defined activities rather than an imperative programming model of writing lines of code.
  3. Workflows allow you to declare business rules that are separated from your code making it easier for you to modify them in the future.
  4. Workflows support different styles of systems with sequential and state machine workflows.
  5. An instance of a workflow can be modified dynamically while running in the event that new conditions require the workflow to behave differently than it did when it was created

Friday, October 24, 2008

Problem- Solution- Pattern Triology

  • Seeking for better raises problems.
  • Problems encourage beautifull minds to find solutions.
  • Solutions may be objects or actions.
  • Objects and actions some times follow patterns, so beatifull minds again encoutaged to discover these patterns.
  • Discovering patterns lead to speed development of other solutions that folow these patterns.
  • But there is no best solution ever so the solution itself can have another problems inside it
  • Applying the solution raise these problems.
  • And we start over again and again

Hints in OOP and design patterns


  • Abstraction is all about breaking your approach to a problem into natural segments. This is where you come up with the objects that divide the problem into manageable parts. In other words, abstracting a problem simply.
  • Abstraction Means thinking of how to tackle that problem in terms of object-oriented code. The data items needed by each object become that object’s properties, whether public or private, and the actions each object needs to perform in the real world become its actions in code.

OOP Principles

  • Objects should represents real life entities.
  • Single responsibility — a class should have only one thing to do.
  • As possible as you can Make your code closed for modification, open for extension.
  • Maintenance should not cost a lot
  • Principle of least know: separate entities (classes of objects) have not to know too much about each other. As much as possible, you should lock away the details inside each class or object and make the coupling between entities as loose as possible If one object needs to know too much about another make their coupling loose (Always go for the loosest coupling you can.).
  • Pure inheritance : inherit every thing

Object relationships

  • Inheritance = "IS A" Relationship
  • Composition = "Has A" Relationship
  • Aggregation = "Has A" Relationship

Design Patterns

  • In problem solving we use the essence of previous problem’s solution to solve the new one (thinking in problem-solution pairs).
    Abstracting from specific problem-solution pairs and distilling out common factors leads to patterns: These problem-solution pairs tend to fall into families of similar problems and solutions with each family exhibiting a pattern in both the problems and the solutions.
    Every pattern deals with a specific, recurring problem in the design or implementation of a software system.
  • Patterns can be used to construct software architectures with specific properties.
  • Design Patterns are solutions for well known programming problems
  • Help making your code closed for modification but open for extension
  • Much of what design patterns are all about has to do with making sure you’re setting up the way you attack the problem correctly. Working with design patterns often means spending more time on the abstraction part of the process than on the concrete classes part.